Jul 28 15:32:51.876375 ip-10-0-134-118 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 15:32:52.267538 ip-10-0-134-118 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:52.267538 ip-10-0-134-118 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 15:32:52.267538 ip-10-0-134-118 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:52.267538 ip-10-0-134-118 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 15:32:52.267538 ip-10-0-134-118 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:52.268378 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.268259    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 15:32:52.272415 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272396    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.272415 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272414    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272419    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272423    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272427    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272430    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272432    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272437    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272442    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272445    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272448    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272451    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272459    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272462    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272466    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272468    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272471    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272473    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272476    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272479    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.272520 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272481    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272484    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272486    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272489    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272492    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272495    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272498    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272501    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272503    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272506    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272508    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272511    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272513    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272516    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272519    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272522    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272525    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272527    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272530    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.272965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272533    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272535    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272538    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272541    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272543    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272546    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272549    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272552    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272554    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272556    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272559    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272561    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272564    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272567    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272570    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272572    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272575    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272577    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272580    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272582    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.273439 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272585    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272587    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272589    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272592    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272594    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272597    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272599    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272602    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272606    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272609    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272614    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272618    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272621    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272624    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272626    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272629    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272631    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272634    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272637    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272639    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.273923 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272642    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272645    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272648    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272650    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272653    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272656    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272658    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272661    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272663    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272666    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272668    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272671    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272673    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272676    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272681    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272684    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272687    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272690    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272694    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.272697    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.274422 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273429    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273437    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273441    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273445    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273448    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273452    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273454    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273457    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273460    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273463    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273465    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273468    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273471    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273474    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273477    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273479    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273482    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273485    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273488    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.274913 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273490    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273493    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273496    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273498    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273501    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273504    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273507    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273509    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273512    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273514    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273517    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273520    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273523    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273526    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273528    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273531    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273534    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273537    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273539    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273542    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.275450 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273544    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273547    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273550    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273552    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273555    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273557    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273563    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273568    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273572    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273576    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273579    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273582    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273585    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273588    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273591    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273594    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273596    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273599    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.275938 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273601    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273604    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273606    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273609    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273612    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273616    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273618    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273621    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273624    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273626    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273629    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273631    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273633    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273636    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273638    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273641    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273643    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273646    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273649    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.276419 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273651    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273654    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273657    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273660    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273663    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273665    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273668    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273670    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273673    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273676    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273679    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273681    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273684    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273687    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273689    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273692    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273694    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273697    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273700    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273702    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.276884 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273705    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.273707    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274076    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274080    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274083    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274086    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274088    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274105    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274109    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274111    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274114    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274118    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274122    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274126    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274129    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274131    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274134    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274137    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274139    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274142    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.277389 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274145    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274148    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274153    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274155    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274165    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274168    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274171    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274174    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274176    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274179    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274181    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274184    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274187    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274191    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274193    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274196    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274199    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274201    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274204    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274206    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.277888 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274209    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274211    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274214    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274217    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274219    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274222    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274225    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274228    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274230    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274233    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274236    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274239    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274241    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274244    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274248    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274251    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274253    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274256    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274258    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.278404 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274261    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274264    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274266    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274269    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274272    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274274    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274277    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274279    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274282    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274284    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274287    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274289    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274292    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274294    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274299    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274303    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274306    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274309    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274311    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.278859 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274314    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274318    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274320    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274323    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274325    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274328    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274330    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274333    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274336    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274339    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274341    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274344    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274346    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274349    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274352    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274354    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274357    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274359    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274362    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274364    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.279348 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274367    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274369    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.274372    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274454    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274464    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274473    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274480    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274485    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274488    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274493    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274498    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274501    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274505    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274508    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274512    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274515    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274519    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274522    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274525    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274527    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274531    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274536    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274539    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274542    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 15:32:52.279932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274545    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274548    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274552    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274555    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274558    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274561    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274565    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274567    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274570    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274574    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274576    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274580    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274583    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274586    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274589    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274592    2577 flags.go:64] FLAG: --enable-server="true"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274595    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274599    2577 flags.go:64] FLAG: --event-burst="100"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274602    2577 flags.go:64] FLAG: --event-qps="50"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274605    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274608    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274611    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274615    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274619    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274622    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 15:32:52.280561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274624    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274628    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274630    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274634    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274637    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274639    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274642    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274645    2577 flags.go:64] FLAG: --feature-gates=""
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274649    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274652    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274656    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274659    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274662    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274664    2577 flags.go:64] FLAG: --help="false"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274668    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274671    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274674    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274677    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274680    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274683    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274686    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274689    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274692    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 15:32:52.281167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274694    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274697    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274700    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274703    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274707    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274710    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274713    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274715    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274719    2577 flags.go:64] FLAG: --lock-file=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274722    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274725    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274728    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274734    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274737    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274739    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274742    2577 flags.go:64] FLAG: --logging-format="text"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274745    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274749    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274751    2577 flags.go:64] FLAG: --manifest-url=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274754    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274758    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274761    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274765    2577 flags.go:64] FLAG: --max-pods="110"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274768    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274771    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 15:32:52.281746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274774    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274828    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274832    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274836    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274839    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274846    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274849    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274852    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274855    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274858    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274864    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274867    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274870    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274873    2577 flags.go:64] FLAG: --port="10250"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274876    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274879    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a064f07711d84abd"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274885    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274888    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274891    2577 flags.go:64] FLAG: --register-node="true"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274894    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274900    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274903    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274905    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274908    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274912    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 15:32:52.282741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274915    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274918    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274921    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274924    2577 flags.go:64] FLAG: --runonce="false"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274927    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274930    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274933    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274936    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274939    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274942    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274945    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274948    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274951    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274953    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274956    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274959    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274964    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274967    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274970    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274975    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274978    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274981    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274985    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274988    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274992    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 15:32:52.283575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274995    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.274998    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275002    2577 flags.go:64] FLAG: --v="2"
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275006    2577 flags.go:64] FLAG: --version="false"
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275010    2577 flags.go:64] FLAG: --vmodule=""
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275014    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275018    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275147    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275151    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275155    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275157    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275160    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275163    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275165    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275168    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275170    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275173    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275176    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275179    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275181    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275184    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275186    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.284224 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275189    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275193    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275196    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275198    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275201    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275203    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275206    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275208    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275211    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275214    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275217    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275219    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275223    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275227    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275231    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275234    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275237    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275239    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275242    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.284793 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275245    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275248    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275251    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275254    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275256    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275259    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275261    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275264    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275267    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275269    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275272    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275275    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275277    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275280    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275284    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275287    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275290    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275292    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275295    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275297    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.285292 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275300    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275303    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275305    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275308    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275310    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275315    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275318    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275320    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275323    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275325    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275328    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275330    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275333    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275336    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275338    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275341    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275343    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275346    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275348    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275351    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.285786 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275353    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275356    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275358    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275361    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275363    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275366    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275370    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275373    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275375    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275378    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275380    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275383    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275385    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275388    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275392    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275395    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275398    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275403    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275406    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.286350 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275409    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275411    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275414    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275418    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275421    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.275424    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.275872    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.283687    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.283704    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.283719    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.283725    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283784    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283789    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283792    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.286822 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283796    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283799    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283802    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283804    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283807    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283810    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283813    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283816    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283818    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283821    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283824    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283827    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283834    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283836    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283839    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283842    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283844    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283847    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283851    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283854    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283857    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.287189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283859    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283862    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283865    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283868    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283870    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283874    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283876    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283879    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283882    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283885    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283887    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283890    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283893    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283896    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283898    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283901    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283903    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283906    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283908    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283911    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.287737 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283913    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283916    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283919    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283922    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283924    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283927    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283929    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283932    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283935    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283938    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283940    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283943    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283945    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283949    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283952    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283954    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283957    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283959    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283962    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283964    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.288243 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283967    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283969    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283972    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283975    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283977    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283979    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283982    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283984    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283988    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283992    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283996    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.283999    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284002    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284005    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284007    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284011    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284014    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284016    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284019    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.288760 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284022    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284026    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284029    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284033    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284035    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284038    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284041    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284044    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284047    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284050    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284052    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284055    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284057    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284060    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284063    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.289244 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284066    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.284071    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284201    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284208    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284211    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284214    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284217    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284219    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284222    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284225    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284228    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284231    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284233    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284236    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284240    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:52.289601 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284243    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284247    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284251    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284253    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284256    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284259    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284262    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284264    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284267    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284270    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284272    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284275    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284278    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284280    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284283    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284285    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284288    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284291    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284294    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:52.289982 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284296    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284299    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284302    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284306    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284310    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284313    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284315    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284318    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284321    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284324    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284327    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284329    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284331    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284335    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284338    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284340    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284343    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284345    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284348    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284350    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:52.290481 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284352    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284355    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284357    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284360    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284363    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284365    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284368    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284370    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284373    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284375    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284378    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284381    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284383    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284386    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284388    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284391    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284393    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284396    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284398    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284401    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:52.290992 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284404    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284407    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284409    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284411    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284414    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284417    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284420    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284423    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284425    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284428    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284431    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284433    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284436    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284438    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284441    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284445    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284448    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284450    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284453    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:52.291634 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284455    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284458    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284461    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284463    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284466    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284469    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284471    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:52.284474    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.284479    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.284604    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.287323    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.288487    2577 server.go:1019] "Starting client certificate rotation"
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.288586    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:32:52.292153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.288627    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:32:52.292484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.290631    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:32:52.292484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.291838    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:32:52.306387 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.306370    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 28 15:32:52.308408 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.308394    2577 log.go:25] "Validated CRI v1 image API"
Jul 28 15:32:52.309419 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.309403    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 15:32:52.314152 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.314106    2577 fs.go:135] Filesystem UUIDs: map[6a6c71ef-531b-4d20-823c-c3a5a1e294d5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 881fa1dd-5523-49e8-8262-737625852b97:/dev/nvme0n1p4]
Jul 28 15:32:52.314227 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.314151    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 15:32:52.320498 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.320376    2577 manager.go:216] Machine: {Timestamp:2026-07-28 15:32:52.318834799 +0000 UTC m=+0.319074718 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090620 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a88ed8dc174195bf0f41d8adb92c SystemUUID:ec28a88e-d8dc-1741-95bf-0f41d8adb92c BootID:832972ae-3139-45c9-ab17-98c5295d7d4d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:65:f9:ac:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:65:f9:ac:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:af:e5:04:8d:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 15:32:52.320498 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.320485    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 15:32:52.320654 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.320565    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 15:32:52.320850 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.320830    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:32:52.323343 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.323309    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 15:32:52.323879 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.323734    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 15:32:52.324958 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.324946    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 15:32:52.325000 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.324961    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 15:32:52.325000 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.324970    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:32:52.325704 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.325693    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:32:52.326428 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.326417    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 15:32:52.328080 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328068    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:32:52.328219 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328209    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 15:32:52.328273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328240    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 15:32:52.328273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328250    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 15:32:52.328273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328262    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 15:32:52.328273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328273    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 28 15:32:52.328439 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.328281    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 15:32:52.329344 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.329320    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:32:52.329460 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.329355    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:32:52.332712 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.332693    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 15:32:52.334634 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.334620    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 15:32:52.334681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.334653    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 15:32:52.336072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336060    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336077    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336084    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336089    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336101    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336109    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336117    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336123    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336130    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 15:32:52.336134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336136    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 15:32:52.336457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336146    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 15:32:52.336457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336151    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 15:32:52.336457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336186    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 15:32:52.336457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.336192    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 15:32:52.339710 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.339697    2577 server.go:1268] "Started kubelet"
Jul 28 15:32:52.339866 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.339817    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 15:32:52.340034 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.339816    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 15:32:52.340177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.340048    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 15:32:52.340687 ip-10-0-134-118 systemd[1]: Started Kubernetes Kubelet.
Jul 28 15:32:52.341924 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.341900    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 15:32:52.343241 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.343198    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 15:32:52.343332 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.343312    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 15:32:52.343409 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.343385    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 15:32:52.343471 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.343446    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 15:32:52.347162 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.347143    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 15:32:52.347503 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.346351    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-118.ec2.internal.18c67df011965896  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-118.ec2.internal,UID:ip-10-0-134-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-118.ec2.internal,},FirstTimestamp:2026-07-28 15:32:52.339669142 +0000 UTC m=+0.339909060,LastTimestamp:2026-07-28 15:32:52.339669142 +0000 UTC m=+0.339909060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-118.ec2.internal,}"
Jul 28 15:32:52.348497 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.348481    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95vr8"
Jul 28 15:32:52.348895 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.348875    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 15:32:52.348987 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.348925    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 15:32:52.348987 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.348981    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.349511 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349491    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 15:32:52.349511 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349491    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 15:32:52.349658 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349520    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 15:32:52.349729 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349693    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 15:32:52.349729 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349705    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 15:32:52.349920 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349902    2577 factory.go:55] Registering systemd factory
Jul 28 15:32:52.349995 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.349924    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 28 15:32:52.350200 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350175    2577 factory.go:153] Registering CRI-O factory
Jul 28 15:32:52.350200 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350192    2577 factory.go:223] Registration of the crio container factory successfully
Jul 28 15:32:52.350311 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350246    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 15:32:52.350311 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350274    2577 factory.go:103] Registering Raw factory
Jul 28 15:32:52.350311 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350292    2577 manager.go:1196] Started watching for new ooms in manager
Jul 28 15:32:52.350991 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.350709    2577 manager.go:349] Starting recovery of all containers
Jul 28 15:32:52.354999 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.354967    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 15:32:52.355201 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.355153    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 15:32:52.357832 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.357802    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95vr8"
Jul 28 15:32:52.361280 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.361258    2577 manager.go:354] Recovery completed
Jul 28 15:32:52.364090 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.364062    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 15:32:52.367408 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.367392    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.369457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.369440    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.369524 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.369477    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.369524 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.369492    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.370121 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370106    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.370223 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370111    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 15:32:52.370223 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370193    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 15:32:52.370223 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370214    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:32:52.370872 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370852    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.370941 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370888    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.370941 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.370903    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.371717 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.371705    2577 policy_none.go:49] "None policy: Start"
Jul 28 15:32:52.371753 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.371721    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 15:32:52.371753 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.371732    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 15:32:52.372224 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.372133    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-118.ec2.internal.18c67df0135ce005  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-118.ec2.internal,UID:ip-10-0-134-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-118.ec2.internal,},FirstTimestamp:2026-07-28 15:32:52.369457157 +0000 UTC m=+0.369697084,LastTimestamp:2026-07-28 15:32:52.369457157 +0000 UTC m=+0.369697084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-118.ec2.internal,}"
Jul 28 15:32:52.372646 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.372635    2577 policy_none.go:47] "Start"
Jul 28 15:32:52.415094 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.414882    2577 manager.go:337] "Starting Device Plugin manager"
Jul 28 15:32:52.415233 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.415138    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 15:32:52.415233 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415155    2577 server.go:87] "Starting device plugin registration server"
Jul 28 15:32:52.415455 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415443    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 15:32:52.415493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415460    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 15:32:52.415579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415558    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 15:32:52.416225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415651    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 15:32:52.416225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.415663    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 15:32:52.416422 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.416298    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 15:32:52.416422 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.416377    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.516318 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.516280    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.519094 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.519013    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.519094 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.519046    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.519094 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.519058    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.519094 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.519088    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.528156 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.528138    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.528247 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.528162    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-118.ec2.internal\": node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.541533 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.541508    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.554373 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.554345    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 15:32:52.555804 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.555782    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 15:32:52.555896 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.555822    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 15:32:52.555896 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.555841    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 15:32:52.555896 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.555879    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 15:32:52.557748 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.557728    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:52.645502 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.645465    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.656383 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.656348    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"]
Jul 28 15:32:52.656476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.656422    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.661210 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.661186    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.661339 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.661223    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.661339 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.661234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.662468 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.662454    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.662599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.662583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.662657 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.662624    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.663305 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663279    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.663305 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663292    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.663448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663314    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.663448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663318    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.663448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663328    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.663448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.663329    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.664507 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.664490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.664612 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.664522    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:52.667389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.667368    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:52.667464 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.667399    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:52.667464 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.667410    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:52.688766 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.688744    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-118.ec2.internal\" not found" node="ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.693448 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.693401    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-118.ec2.internal\" not found" node="ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.745901 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.745869    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.751215 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.751193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.751344 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.751231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/389edd964b978460b3dd3a4bae70f4b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-118.ec2.internal\" (UID: \"389edd964b978460b3dd3a4bae70f4b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.751344 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.751256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.846596 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.846485    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.852260 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.852322 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.852322 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/389edd964b978460b3dd3a4bae70f4b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-118.ec2.internal\" (UID: \"389edd964b978460b3dd3a4bae70f4b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.852427 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/389edd964b978460b3dd3a4bae70f4b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-118.ec2.internal\" (UID: \"389edd964b978460b3dd3a4bae70f4b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.852427 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.852427 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.852381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73911d6eb94615343d83a506c167c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal\" (UID: \"f73911d6eb94615343d83a506c167c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.954272 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:52.954238    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:52.993370 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.993342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:52.997369 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:52.997164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:53.055237 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:53.055207    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:53.155596 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:53.155522    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:53.255647 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:53.255617    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:53.288862 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.288826    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 15:32:53.289333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.288991    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:32:53.356616 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:53.356587    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-118.ec2.internal\" not found"
Jul 28 15:32:53.363401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.363369    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:27:53 +0000 UTC" deadline="2028-03-19 20:17:44.05674609 +0000 UTC"
Jul 28 15:32:53.363401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.363399    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14404h44m50.693350085s"
Jul 28 15:32:53.372919 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.372900    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:53.403444 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.403421    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:53.449525 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.449494    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:53.462924 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.462893    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:32:53.465483 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.465466    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal"
Jul 28 15:32:53.477215 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.477120    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:32:53.523730 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:53.523698    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389edd964b978460b3dd3a4bae70f4b1.slice/crio-480222a94d186a3feb32f4aba028a230c049755e52228ba6fbdd6b535de22779 WatchSource:0}: Error finding container 480222a94d186a3feb32f4aba028a230c049755e52228ba6fbdd6b535de22779: Status 404 returned error can't find the container with id 480222a94d186a3feb32f4aba028a230c049755e52228ba6fbdd6b535de22779
Jul 28 15:32:53.525697 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:53.525655    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73911d6eb94615343d83a506c167c3d.slice/crio-b677851e673ae74068e72acd36abf10757f44e0645f337a8cebc7e443fb54ae0 WatchSource:0}: Error finding container b677851e673ae74068e72acd36abf10757f44e0645f337a8cebc7e443fb54ae0: Status 404 returned error can't find the container with id b677851e673ae74068e72acd36abf10757f44e0645f337a8cebc7e443fb54ae0
Jul 28 15:32:53.530251 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.530231    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:32:53.559862 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.559803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal" event={"ID":"389edd964b978460b3dd3a4bae70f4b1","Type":"ContainerStarted","Data":"480222a94d186a3feb32f4aba028a230c049755e52228ba6fbdd6b535de22779"}
Jul 28 15:32:53.561046 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.561018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal" event={"ID":"f73911d6eb94615343d83a506c167c3d","Type":"ContainerStarted","Data":"b677851e673ae74068e72acd36abf10757f44e0645f337a8cebc7e443fb54ae0"}
Jul 28 15:32:53.563559 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:53.563534    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:54.331523 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.331479    2577 apiserver.go:52] "Watching apiserver"
Jul 28 15:32:54.342250 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.342217    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 15:32:54.343927 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.343877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal","openshift-cluster-node-tuning-operator/tuned-grk6l","openshift-dns/node-resolver-df56s","openshift-image-registry/node-ca-9hgcb","openshift-multus/multus-v2pc9","openshift-network-operator/iptables-alerter-bfj4g","kube-system/konnectivity-agent-2p9tk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal","openshift-multus/multus-additional-cni-plugins-fmx6g","openshift-multus/network-metrics-daemon-tk25s","openshift-network-diagnostics/network-check-target-wmd7f","openshift-ovn-kubernetes/ovnkube-node-zcw4d","kube-system/global-pull-secret-syncer-2b2bp"]
Jul 28 15:32:54.345783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.345757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.346933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.346906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.348627 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.348603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.349130 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.349108    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rbpc\""
Jul 28 15:32:54.349382 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.349364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 15:32:54.349593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.349575    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 15:32:54.350532 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.350508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p2xp7\""
Jul 28 15:32:54.350532 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.350519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 15:32:54.350771 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.350752    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 15:32:54.350983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.350959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.351045 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.351004    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 15:32:54.351226 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.351208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 15:32:54.351698 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.351677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.352769 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.352748    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.352982 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.352963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.353262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2ttjn\""
Jul 28 15:32:54.353340 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353282    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 15:32:54.353397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353388    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.353506 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353476    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 15:32:54.353607 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353589    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.353831 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.353913 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.353898    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.354091 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.354067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67rnw\""
Jul 28 15:32:54.354329 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.354309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.354419 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.354379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:32:54.354542 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.354523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.354878 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.354855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.355296 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.355278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.355484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.355468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 15:32:54.355689 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.355673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 15:32:54.355841 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.355824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.355903 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.355893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7zj8r\""
Jul 28 15:32:54.356840 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.356820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.357665 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.357642    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.357767 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.357725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dz8v\""
Jul 28 15:32:54.358169 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.358150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.361095 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.360879    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2dj48\""
Jul 28 15:32:54.361158 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.361116    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.361566 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.361541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.361732 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.361707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 15:32:54.362674 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.362655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 15:32:54.362889 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.362871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.362963 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.362932    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 15:32:54.363023 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.362992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 15:32:54.363086 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 15:32:54.363460 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.363594 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.363570    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:32:54.363702 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vcb52\""
Jul 28 15:32:54.363918 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-node-log\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.363969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-env-overrides\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-systemd-units\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-etc-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmwt\" (UniqueName: \"kubernetes.io/projected/a03cfd2f-05a8-4157-87c4-8dc38753fede-kube-api-access-rrmwt\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f84d86c-9a2d-432c-b154-72723a223f65-konnectivity-ca\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-netd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0de70cee-1194-44a8-b74a-420221243eb4-ovn-node-metrics-cert\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-slash\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-netns\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-ovn\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr77w\""
Jul 28 15:32:54.364777 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.364770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364788    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbd0202-abf5-4432-8dc4-1197b601c3e6-tmp-dir\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-kubelet\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-var-lib-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-bin\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a03cfd2f-05a8-4157-87c4-8dc38753fede-serviceca\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.364991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-log-socket\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-script-lib\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdf5\" (UniqueName: \"kubernetes.io/projected/0de70cee-1194-44a8-b74a-420221243eb4-kube-api-access-6pdf5\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdbd0202-abf5-4432-8dc4-1197b601c3e6-hosts-file\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a03cfd2f-05a8-4157-87c4-8dc38753fede-host\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-systemd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-config\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f84d86c-9a2d-432c-b154-72723a223f65-agent-certs\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.365851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.365330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckws\" (UniqueName: \"kubernetes.io/projected/cdbd0202-abf5-4432-8dc4-1197b601c3e6-kube-api-access-pckws\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.372066 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.370397    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:32:54.394465 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.394406    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-976mq"
Jul 28 15:32:54.401365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.401338    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-976mq"
Jul 28 15:32:54.451395 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.451370    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 15:32:54.466072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ea6d915-74fe-46a4-a2b8-921984137503-host-slash\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.466072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-kubelet\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-multus-certs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ea6d915-74fe-46a4-a2b8-921984137503-iptables-alerter-script\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-dbus\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-netns\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysconfig\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-sys\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.466304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-etc-tuned\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p8q\" (UniqueName: \"kubernetes.io/projected/1829f02b-02ea-4653-bf43-2cf781f89627-kube-api-access-d8p8q\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f84d86c-9a2d-432c-b154-72723a223f65-agent-certs\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-kubelet-config\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-conf-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8fs\" (UniqueName: \"kubernetes.io/projected/a3716281-5742-4fe4-8a3d-36ff4b977445-kube-api-access-7l8fs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-node-log\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-etc-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmwt\" (UniqueName: \"kubernetes.io/projected/a03cfd2f-05a8-4157-87c4-8dc38753fede-kube-api-access-rrmwt\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc29\" (UniqueName: \"kubernetes.io/projected/eafbb7e1-e47c-429d-90df-3165977d0ae8-kube-api-access-mpc29\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.466649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f84d86c-9a2d-432c-b154-72723a223f65-konnectivity-ca\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0de70cee-1194-44a8-b74a-420221243eb4-ovn-node-metrics-cert\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-var-lib-kubelet\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-slash\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-ovn\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-multus\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-conf\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-lib-modules\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a03cfd2f-05a8-4157-87c4-8dc38753fede-serviceca\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-kube-api-access-tdn94\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-cnibin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-cni-binary-copy\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.467198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-hostroot\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.466998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-device-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-system-cni-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skd67\" (UniqueName: \"kubernetes.io/projected/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kube-api-access-skd67\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-kubernetes\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-systemd\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-systemd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-config\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-ovn\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467685    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckws\" (UniqueName: \"kubernetes.io/projected/cdbd0202-abf5-4432-8dc4-1197b601c3e6-kube-api-access-pckws\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-system-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-env-overrides\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468092 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-os-release\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a03cfd2f-05a8-4157-87c4-8dc38753fede-serviceca\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-systemd-units\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-bin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-modprobe-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-run\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f84d86c-9a2d-432c-b154-72723a223f65-konnectivity-ca\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-node-log\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-etc-kubernetes\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-netd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-os-release\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-k8s-cni-cncf-io\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-netd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-host\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.468985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-etc-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-tmp\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-netns\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-systemd\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-env-overrides\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbd0202-abf5-4432-8dc4-1197b601c3e6-tmp-dir\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-run-netns\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-slash\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-systemd-units\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.467944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-run-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6qw\" (UniqueName: \"kubernetes.io/projected/5ea6d915-74fe-46a4-a2b8-921984137503-kube-api-access-7c6qw\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbd0202-abf5-4432-8dc4-1197b601c3e6-tmp-dir\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-daemon-config\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-kubelet\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-var-lib-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-kubelet\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.469936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-bin\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-config\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-host-cni-bin\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-socket-dir-parent\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-var-lib-openvswitch\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-cnibin\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.468969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-log-socket\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-script-lib\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0de70cee-1194-44a8-b74a-420221243eb4-log-socket\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdf5\" (UniqueName: \"kubernetes.io/projected/0de70cee-1194-44a8-b74a-420221243eb4-kube-api-access-6pdf5\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdbd0202-abf5-4432-8dc4-1197b601c3e6-hosts-file\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdbd0202-abf5-4432-8dc4-1197b601c3e6-hosts-file\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a03cfd2f-05a8-4157-87c4-8dc38753fede-host\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a03cfd2f-05a8-4157-87c4-8dc38753fede-host\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.470809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.469571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0de70cee-1194-44a8-b74a-420221243eb4-ovnkube-script-lib\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.471988 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.471965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0de70cee-1194-44a8-b74a-420221243eb4-ovn-node-metrics-cert\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.472089 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.472015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f84d86c-9a2d-432c-b154-72723a223f65-agent-certs\") pod \"konnectivity-agent-2p9tk\" (UID: \"8f84d86c-9a2d-432c-b154-72723a223f65\") " pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.484485 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.484408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdf5\" (UniqueName: \"kubernetes.io/projected/0de70cee-1194-44a8-b74a-420221243eb4-kube-api-access-6pdf5\") pod \"ovnkube-node-zcw4d\" (UID: \"0de70cee-1194-44a8-b74a-420221243eb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.485369 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.485211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmwt\" (UniqueName: \"kubernetes.io/projected/a03cfd2f-05a8-4157-87c4-8dc38753fede-kube-api-access-rrmwt\") pod \"node-ca-9hgcb\" (UID: \"a03cfd2f-05a8-4157-87c4-8dc38753fede\") " pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.485683 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.485638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckws\" (UniqueName: \"kubernetes.io/projected/cdbd0202-abf5-4432-8dc4-1197b601c3e6-kube-api-access-pckws\") pod \"node-resolver-df56s\" (UID: \"cdbd0202-abf5-4432-8dc4-1197b601c3e6\") " pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.569589 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-multus\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-conf\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-lib-modules\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-multus\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-kube-api-access-tdn94\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-cnibin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.569751 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-cni-binary-copy\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-hostroot\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-hostroot\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-lib-modules\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-cnibin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-conf\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-device-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-system-cni-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.569988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-device-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-system-cni-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skd67\" (UniqueName: \"kubernetes.io/projected/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kube-api-access-skd67\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-kubernetes\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-systemd\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-kubernetes\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.570430 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-systemd\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-system-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-os-release\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-system-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-os-release\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-cni-binary-copy\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.570456    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-bin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-modprobe-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-cni-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.570537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:55.070516062 +0000 UTC m=+3.070755983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-cni-bin\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-run\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-modprobe-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.571299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-run\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-etc-kubernetes\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-os-release\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-k8s-cni-cncf-io\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-host\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-etc-kubernetes\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-tmp\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-k8s-cni-cncf-io\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.570988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6qw\" (UniqueName: \"kubernetes.io/projected/5ea6d915-74fe-46a4-a2b8-921984137503-kube-api-access-7c6qw\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-daemon-config\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-os-release\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-socket-dir-parent\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbb7e1-e47c-429d-90df-3165977d0ae8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-cnibin\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-cnibin\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.572163 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ea6d915-74fe-46a4-a2b8-921984137503-host-slash\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-host\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-kubelet\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-multus-certs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ea6d915-74fe-46a4-a2b8-921984137503-iptables-alerter-script\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-dbus\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-socket-dir-parent\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-netns\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-netns\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysconfig\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-sys\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysconfig\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-etc-tuned\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p8q\" (UniqueName: \"kubernetes.io/projected/1829f02b-02ea-4653-bf43-2cf781f89627-kube-api-access-d8p8q\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-kubelet-config\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-conf-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8fs\" (UniqueName: \"kubernetes.io/projected/a3716281-5742-4fe4-8a3d-36ff4b977445-kube-api-access-7l8fs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-daemon-config\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-var-lib-kubelet\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc29\" (UniqueName: \"kubernetes.io/projected/eafbb7e1-e47c-429d-90df-3165977d0ae8-kube-api-access-mpc29\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-kubelet-config\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/019a3992-b0cd-4983-b0b6-57510bb4458c-dbus\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-var-lib-kubelet\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ea6d915-74fe-46a4-a2b8-921984137503-iptables-alerter-script\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-multus-conf-dir\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.571750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ea6d915-74fe-46a4-a2b8-921984137503-host-slash\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.573910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbb7e1-e47c-429d-90df-3165977d0ae8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.572321    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3716281-5742-4fe4-8a3d-36ff4b977445-host-run-multus-certs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-var-lib-kubelet\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.572437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:55.07241854 +0000 UTC m=+3.072658460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-sys\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1829f02b-02ea-4653-bf43-2cf781f89627-etc-sysctl-d\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.574868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.572848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d95b1e3-d237-49d3-b3b0-076f7cf09367-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.575286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.574949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-etc-tuned\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.575286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.575003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1829f02b-02ea-4653-bf43-2cf781f89627-tmp\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.590761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.590653    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:54.590761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.590684    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:54.590761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.590697    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:54.590972 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:54.590838    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:55.090817198 +0000 UTC m=+3.091057124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:54.592102 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.592068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-kube-api-access-tdn94\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:54.592413 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.592382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc29\" (UniqueName: \"kubernetes.io/projected/eafbb7e1-e47c-429d-90df-3165977d0ae8-kube-api-access-mpc29\") pod \"multus-additional-cni-plugins-fmx6g\" (UID: \"eafbb7e1-e47c-429d-90df-3165977d0ae8\") " pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.594060 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.594011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p8q\" (UniqueName: \"kubernetes.io/projected/1829f02b-02ea-4653-bf43-2cf781f89627-kube-api-access-d8p8q\") pod \"tuned-grk6l\" (UID: \"1829f02b-02ea-4653-bf43-2cf781f89627\") " pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.594811 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.594780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8fs\" (UniqueName: \"kubernetes.io/projected/a3716281-5742-4fe4-8a3d-36ff4b977445-kube-api-access-7l8fs\") pod \"multus-v2pc9\" (UID: \"a3716281-5742-4fe4-8a3d-36ff4b977445\") " pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.595718 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.595375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6qw\" (UniqueName: \"kubernetes.io/projected/5ea6d915-74fe-46a4-a2b8-921984137503-kube-api-access-7c6qw\") pod \"iptables-alerter-bfj4g\" (UID: \"5ea6d915-74fe-46a4-a2b8-921984137503\") " pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.596051 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.596016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skd67\" (UniqueName: \"kubernetes.io/projected/9d95b1e3-d237-49d3-b3b0-076f7cf09367-kube-api-access-skd67\") pod \"aws-ebs-csi-driver-node-5bkl6\" (UID: \"9d95b1e3-d237-49d3-b3b0-076f7cf09367\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.668251 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.668201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:32:54.676749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.676720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:32:54.686061 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.686034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-df56s"
Jul 28 15:32:54.692285 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.692262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hgcb"
Jul 28 15:32:54.701773 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.701750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2pc9"
Jul 28 15:32:54.712222 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.712200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bfj4g"
Jul 28 15:32:54.729675 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.729649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6"
Jul 28 15:32:54.735240 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.735219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grk6l"
Jul 28 15:32:54.739824 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.739804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fmx6g"
Jul 28 15:32:54.740127 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:54.740101    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:55.076197 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.076158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:55.076391 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.076212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:55.076391 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.076327    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:55.076488 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.076407    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:56.076380951 +0000 UTC m=+4.076620860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:55.076488 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.076327    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:55.076488 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.076466    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:56.076451665 +0000 UTC m=+4.076691578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:55.177295 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.177251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:55.177493 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.177412    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:55.177493 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.177434    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:55.177493 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.177448    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:55.177683 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.177513    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:56.177494361 +0000 UTC m=+4.177734270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:55.405114 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.405003    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:27:55 +0000 UTC" deadline="2027-12-31 09:26:48.257264315 +0000 UTC"
Jul 28 15:32:55.405114 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.405039    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12497h53m52.852228186s"
Jul 28 15:32:55.556436 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.556389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:55.556609 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.556440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:55.556609 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:55.556514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:55.556609 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.556532    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:32:55.556794 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.556608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:32:55.556794 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:55.556679    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:32:55.639825 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.639783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbd0202_abf5_4432_8dc4_1197b601c3e6.slice/crio-ca4a7c92b9feeaf5b0c6a35249b5276f02c6654342e3f490b30e6df8126eecb0 WatchSource:0}: Error finding container ca4a7c92b9feeaf5b0c6a35249b5276f02c6654342e3f490b30e6df8126eecb0: Status 404 returned error can't find the container with id ca4a7c92b9feeaf5b0c6a35249b5276f02c6654342e3f490b30e6df8126eecb0
Jul 28 15:32:55.663369 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.663339    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f84d86c_9a2d_432c_b154_72723a223f65.slice/crio-01756521348a35be658db46f6834915f8202bcbd99dc692adf4a2a439bed2a71 WatchSource:0}: Error finding container 01756521348a35be658db46f6834915f8202bcbd99dc692adf4a2a439bed2a71: Status 404 returned error can't find the container with id 01756521348a35be658db46f6834915f8202bcbd99dc692adf4a2a439bed2a71
Jul 28 15:32:55.663639 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.663622    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d95b1e3_d237_49d3_b3b0_076f7cf09367.slice/crio-46c9e37cc6f9dc435c3e7915dd5c2220ea5d3668eb71e7104e456439f52cf50f WatchSource:0}: Error finding container 46c9e37cc6f9dc435c3e7915dd5c2220ea5d3668eb71e7104e456439f52cf50f: Status 404 returned error can't find the container with id 46c9e37cc6f9dc435c3e7915dd5c2220ea5d3668eb71e7104e456439f52cf50f
Jul 28 15:32:55.668082 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.668060    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1829f02b_02ea_4653_bf43_2cf781f89627.slice/crio-810d3d256e51a6c35f85b980fa460be638105075d9995e11a28d141501fdcde5 WatchSource:0}: Error finding container 810d3d256e51a6c35f85b980fa460be638105075d9995e11a28d141501fdcde5: Status 404 returned error can't find the container with id 810d3d256e51a6c35f85b980fa460be638105075d9995e11a28d141501fdcde5
Jul 28 15:32:55.669086 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.669047    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03cfd2f_05a8_4157_87c4_8dc38753fede.slice/crio-d534be6257222d38ef4851caccd9dcd5cb9dc4d4c8e5baaa642ac4cecf82f082 WatchSource:0}: Error finding container d534be6257222d38ef4851caccd9dcd5cb9dc4d4c8e5baaa642ac4cecf82f082: Status 404 returned error can't find the container with id d534be6257222d38ef4851caccd9dcd5cb9dc4d4c8e5baaa642ac4cecf82f082
Jul 28 15:32:55.674538 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.674317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3716281_5742_4fe4_8a3d_36ff4b977445.slice/crio-3533e5a7837dbed3c756b4c4727edebe0db3f3507e3a1570311d6ad9fb25de44 WatchSource:0}: Error finding container 3533e5a7837dbed3c756b4c4727edebe0db3f3507e3a1570311d6ad9fb25de44: Status 404 returned error can't find the container with id 3533e5a7837dbed3c756b4c4727edebe0db3f3507e3a1570311d6ad9fb25de44
Jul 28 15:32:55.674609 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:32:55.674578    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea6d915_74fe_46a4_a2b8_921984137503.slice/crio-4e3b6987cd725c3acb8c84dfad7bf9567a6673c81210ac01e20821114a67f148 WatchSource:0}: Error finding container 4e3b6987cd725c3acb8c84dfad7bf9567a6673c81210ac01e20821114a67f148: Status 404 returned error can't find the container with id 4e3b6987cd725c3acb8c84dfad7bf9567a6673c81210ac01e20821114a67f148
Jul 28 15:32:56.086504 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.086447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:56.086679 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.086533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:56.086771 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.086682    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:56.086771 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.086760    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:58.086739518 +0000 UTC m=+6.086979444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:56.086886 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.086682    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:56.086886 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.086829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:58.086815668 +0000 UTC m=+6.087055578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:56.187465 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.187386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:56.187664 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.187591    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:56.187664 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.187612    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:56.187664 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.187625    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:56.187844 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:56.187689    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:32:58.187668524 +0000 UTC m=+6.187908444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:56.405301 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.405223    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:27:55 +0000 UTC" deadline="2028-04-30 11:53:09.94364398 +0000 UTC"
Jul 28 15:32:56.405301 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.405291    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15404h20m13.538357562s"
Jul 28 15:32:56.575846 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.575745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grk6l" event={"ID":"1829f02b-02ea-4653-bf43-2cf781f89627","Type":"ContainerStarted","Data":"810d3d256e51a6c35f85b980fa460be638105075d9995e11a28d141501fdcde5"}
Jul 28 15:32:56.587591 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.587501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2p9tk" event={"ID":"8f84d86c-9a2d-432c-b154-72723a223f65","Type":"ContainerStarted","Data":"01756521348a35be658db46f6834915f8202bcbd99dc692adf4a2a439bed2a71"}
Jul 28 15:32:56.595418 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.594825    2577 generic.go:356] "Generic (PLEG): container finished" podID="f73911d6eb94615343d83a506c167c3d" containerID="9c694267cebe84ede256431b22fbba345c638171f10b46f4de1de31b87b2f18e" exitCode=0
Jul 28 15:32:56.595418 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.594907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal" event={"ID":"f73911d6eb94615343d83a506c167c3d","Type":"ContainerDied","Data":"9c694267cebe84ede256431b22fbba345c638171f10b46f4de1de31b87b2f18e"}
Jul 28 15:32:56.607515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.607428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bfj4g" event={"ID":"5ea6d915-74fe-46a4-a2b8-921984137503","Type":"ContainerStarted","Data":"4e3b6987cd725c3acb8c84dfad7bf9567a6673c81210ac01e20821114a67f148"}
Jul 28 15:32:56.610125 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.610029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"fab96ab6515943bcc799254e65cd7aa96f9f92ba80059ea3617486b595944560"}
Jul 28 15:32:56.612746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.612686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerStarted","Data":"e9bcf23b78d958b7476c56e2cdda211b6dc2383e98721eb21256cabacf7d5db8"}
Jul 28 15:32:56.629447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.629137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6" event={"ID":"9d95b1e3-d237-49d3-b3b0-076f7cf09367","Type":"ContainerStarted","Data":"46c9e37cc6f9dc435c3e7915dd5c2220ea5d3668eb71e7104e456439f52cf50f"}
Jul 28 15:32:56.649293 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.649259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-df56s" event={"ID":"cdbd0202-abf5-4432-8dc4-1197b601c3e6","Type":"ContainerStarted","Data":"ca4a7c92b9feeaf5b0c6a35249b5276f02c6654342e3f490b30e6df8126eecb0"}
Jul 28 15:32:56.670264 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.669652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal" event={"ID":"389edd964b978460b3dd3a4bae70f4b1","Type":"ContainerStarted","Data":"45458f3478ed63dc60ba50df21baf2bfe3b48f342b161f94dfbc48b32ef5c72c"}
Jul 28 15:32:56.688548 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.688491    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-118.ec2.internal" podStartSLOduration=2.688471304 podStartE2EDuration="2.688471304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:32:56.687460843 +0000 UTC m=+4.687700776" watchObservedRunningTime="2026-07-28 15:32:56.688471304 +0000 UTC m=+4.688711233"
Jul 28 15:32:56.699753 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.699655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2pc9" event={"ID":"a3716281-5742-4fe4-8a3d-36ff4b977445","Type":"ContainerStarted","Data":"3533e5a7837dbed3c756b4c4727edebe0db3f3507e3a1570311d6ad9fb25de44"}
Jul 28 15:32:56.723712 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:56.723628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hgcb" event={"ID":"a03cfd2f-05a8-4157-87c4-8dc38753fede","Type":"ContainerStarted","Data":"d534be6257222d38ef4851caccd9dcd5cb9dc4d4c8e5baaa642ac4cecf82f082"}
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:57.556957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:57.557128    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:57.557552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:57.557645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:57.557701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:57.557828 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:57.557779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:32:57.749288 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:57.749219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal" event={"ID":"f73911d6eb94615343d83a506c167c3d","Type":"ContainerStarted","Data":"12d54e1ad50dc17f00a01a81220984a3379276e762dd88453ace9ec351de2d49"}
Jul 28 15:32:58.110165 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:58.109923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:58.110165 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:58.110020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:58.110465 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.110179    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:58.110465 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.110250    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:02.110229415 +0000 UTC m=+10.110469343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:58.110465 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.110326    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:58.110465 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.110362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:02.11035007 +0000 UTC m=+10.110589981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:58.210447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:58.210408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:58.210654 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.210632    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:58.210719 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.210658    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:58.210719 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.210675    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:58.210820 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:58.210738    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:02.210717871 +0000 UTC m=+10.210957780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:59.556908 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:59.556871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:32:59.557433 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:59.557010    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:32:59.557570 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:59.557547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:32:59.557681 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:59.557658    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:32:59.557748 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:32:59.557737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:32:59.557854 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:32:59.557834    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:01.556129 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:01.556091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:01.556645 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:01.556131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:01.556645 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:01.556091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:01.556645 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:01.556246    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:01.556645 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:01.556324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:01.556645 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:01.556438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:02.149113 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:02.149058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:02.149294 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:02.149144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:02.149294 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.149248    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:02.149418 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.149341    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:10.149318044 +0000 UTC m=+18.149557977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:02.149418 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.149259    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:02.149505 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.149421    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:10.149398703 +0000 UTC m=+18.149638627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:02.249985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:02.249816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:02.250221 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.250022    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:02.250221 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.250054    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:02.250221 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.250070    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:02.250221 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:02.250155    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:10.250133947 +0000 UTC m=+18.250373875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:03.556601 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:03.556560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:03.557076 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:03.556662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:03.557076 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:03.556682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:03.557076 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:03.556791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:03.557076 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:03.556929    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:03.557076 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:03.557010    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:05.556166 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:05.556132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:05.556166 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:05.556163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:05.556673 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:05.556162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:05.556673 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:05.556245    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:05.556673 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:05.556355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:05.556673 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:05.556471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:07.556493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:07.556457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:07.556925 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:07.556565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:07.556925 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:07.556577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:07.556925 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:07.556602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:07.556925 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:07.556680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:07.556925 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:07.556805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:09.556778 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:09.556737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:09.556778 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:09.556778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:09.557299 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:09.556778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:09.557299 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:09.556874    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:09.557299 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:09.556979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:09.557299 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:09.557055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:10.207549 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:10.207511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:10.207699 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:10.207569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:10.207699 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.207665    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:10.207769 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.207684    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:10.207769 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.207732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:26.207714853 +0000 UTC m=+34.207954769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:10.207769 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.207748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:26.207741454 +0000 UTC m=+34.207981358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:10.308284 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:10.308241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:10.308461 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.308412    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:10.308461 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.308435    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:10.308461 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.308449    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:10.308675 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:10.308507    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:26.308486823 +0000 UTC m=+34.308726751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:11.556986 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:11.556948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:11.557476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:11.556948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:11.557476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:11.556948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:11.557476 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:11.557126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:11.557476 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:11.557174    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:11.557476 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:11.557263    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:13.556487 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:13.556446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:13.556984 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:13.556490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:13.556984 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:13.556456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:13.556984 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:13.556598    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:13.556984 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:13.556664    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:13.556984 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:13.556746    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:14.795752 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.795531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2pc9" event={"ID":"a3716281-5742-4fe4-8a3d-36ff4b977445","Type":"ContainerStarted","Data":"0752b122ddf5ae3392a3399ee5b6587cdc5a3a03d5c88cda1e9c7120c2703df8"}
Jul 28 15:33:14.797007 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.796965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hgcb" event={"ID":"a03cfd2f-05a8-4157-87c4-8dc38753fede","Type":"ContainerStarted","Data":"2cfb5237c51af2692842a5b28790d7ecbca08d0aee00ff338313815f7b6a4717"}
Jul 28 15:33:14.802032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.802001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grk6l" event={"ID":"1829f02b-02ea-4653-bf43-2cf781f89627","Type":"ContainerStarted","Data":"041c5c90ef3ba0ea491e14ce42381126380dd27570aaa103459526f07a029f71"}
Jul 28 15:33:14.803524 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.803501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2p9tk" event={"ID":"8f84d86c-9a2d-432c-b154-72723a223f65","Type":"ContainerStarted","Data":"1cb3500c97d10d3bf044ae623cddcae92cccea531c742334f6b30feacef2f6b1"}
Jul 28 15:33:14.809317 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.809298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:33:14.809621 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.809601    2577 generic.go:356] "Generic (PLEG): container finished" podID="0de70cee-1194-44a8-b74a-420221243eb4" containerID="ceb033df627b282032b9ef0a344becdfe11cc1d0251ba6b8ae7404e7df70d90c" exitCode=1
Jul 28 15:33:14.809725 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.809659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerDied","Data":"ceb033df627b282032b9ef0a344becdfe11cc1d0251ba6b8ae7404e7df70d90c"}
Jul 28 15:33:14.809777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.809765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"9cbb3a14763cea41ba56e52e18fa80edf3ec5e581f4aa7fa3a09cedfcbaa1cbd"}
Jul 28 15:33:14.811420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.811395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerStarted","Data":"e8f7091a3f0fc64975a03644b2b3a6294b3ff26966e8a35583d2d68bd8532d4c"}
Jul 28 15:33:14.812946 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.812919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6" event={"ID":"9d95b1e3-d237-49d3-b3b0-076f7cf09367","Type":"ContainerStarted","Data":"ca92321f1301aad226c4207175d0744a9512d5ef00d477ea81bd65d22c95fcd4"}
Jul 28 15:33:14.814470 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.814445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-df56s" event={"ID":"cdbd0202-abf5-4432-8dc4-1197b601c3e6","Type":"ContainerStarted","Data":"75e53a58c4e423d64a8468a13aa4b777df0ca175efea0448e1ec99903ed108ef"}
Jul 28 15:33:14.817391 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.817358    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v2pc9" podStartSLOduration=2.86608667 podStartE2EDuration="21.817346342s" totalImagesPullingTime="18.951259672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:14.816826484 +0000 UTC m=+22.817066409" watchObservedRunningTime="2026-07-28 15:33:14.817346342 +0000 UTC m=+22.817586267"
Jul 28 15:33:14.817576 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.817557    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-118.ec2.internal" podStartSLOduration=20.817550568 podStartE2EDuration="20.817550568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:32:57.76939593 +0000 UTC m=+5.769635860" watchObservedRunningTime="2026-07-28 15:33:14.817550568 +0000 UTC m=+22.817790491"
Jul 28 15:33:14.833579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.833539    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-df56s" podStartSLOduration=3.091497817 podStartE2EDuration="21.833524184s" totalImagesPullingTime="18.742026367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:14.833017058 +0000 UTC m=+22.833256986" watchObservedRunningTime="2026-07-28 15:33:14.833524184 +0000 UTC m=+22.833764109"
Jul 28 15:33:14.875245 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.875141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hgcb" podStartSLOduration=3.144703942 podStartE2EDuration="21.875117551s" totalImagesPullingTime="18.730413609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:14.850345195 +0000 UTC m=+22.850585135" watchObservedRunningTime="2026-07-28 15:33:14.875117551 +0000 UTC m=+22.875357468"
Jul 28 15:33:14.894010 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.893954    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2p9tk" podStartSLOduration=7.491519178 podStartE2EDuration="21.893930988s" totalImagesPullingTime="14.40241181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:14.893308339 +0000 UTC m=+22.893548266" watchObservedRunningTime="2026-07-28 15:33:14.893930988 +0000 UTC m=+22.894170917"
Jul 28 15:33:14.912793 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:14.912746    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-grk6l" podStartSLOduration=3.157992702 podStartE2EDuration="21.912730214s" totalImagesPullingTime="18.754737512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:14.912476343 +0000 UTC m=+22.912716271" watchObservedRunningTime="2026-07-28 15:33:14.912730214 +0000 UTC m=+22.912970138"
Jul 28 15:33:15.142749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.142451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:33:15.142915 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.142900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:33:15.556274 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.556233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:15.556473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.556239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:15.556473 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:15.556371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:15.556588 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:15.556492    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:15.556588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.556252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:15.556688 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:15.556653    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:15.820300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.820271    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:33:15.820901 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.820760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"5bcbe7bca378277dfc9a51d8e0d8904379669edfa2ec11ed60fc4917f4c3974d"}
Jul 28 15:33:15.820901 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.820845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"785750e8bf483812a1f8ec55d10bc5e4124c55fe159cbf5a146c104079928c2e"}
Jul 28 15:33:15.821018 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.820904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"908b6df229f94feff27a2457799bd8ef8283477721939e22c4fda0b30250fede"}
Jul 28 15:33:15.821018 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.820938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"fc7538cf99d58343431577f0890eb4be424f911ddc2d037d747c755bd0062775"}
Jul 28 15:33:15.822273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.822239    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="e8f7091a3f0fc64975a03644b2b3a6294b3ff26966e8a35583d2d68bd8532d4c" exitCode=0
Jul 28 15:33:15.822406 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.822372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"e8f7091a3f0fc64975a03644b2b3a6294b3ff26966e8a35583d2d68bd8532d4c"}
Jul 28 15:33:15.823067 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.823047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:33:15.823534 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:15.823514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2p9tk"
Jul 28 15:33:16.224786 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.224760    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 15:33:16.429580 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.429398    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:33:16.224780768Z","UUID":"2a66dc53-4124-410b-983c-d35ad7e6736b","Handler":null,"Name":"","Endpoint":""}
Jul 28 15:33:16.431065 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.431032    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 15:33:16.431200 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.431079    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 15:33:16.827295 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.827261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6" event={"ID":"9d95b1e3-d237-49d3-b3b0-076f7cf09367","Type":"ContainerStarted","Data":"3d5c4c00d3546c2be23cb7620e44b079be81a18ae8f4faa313b1f17e0eb662ff"}
Jul 28 15:33:16.829399 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.829368    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bfj4g" event={"ID":"5ea6d915-74fe-46a4-a2b8-921984137503","Type":"ContainerStarted","Data":"f2f65353c8d470a7501f4717d9804313beb48f3b576906d072cd88233c248ace"}
Jul 28 15:33:16.849909 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:16.849853    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bfj4g" podStartSLOduration=5.103560232 podStartE2EDuration="23.849831309s" totalImagesPullingTime="18.746271077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:16.849765835 +0000 UTC m=+24.850005786" watchObservedRunningTime="2026-07-28 15:33:16.849831309 +0000 UTC m=+24.850071234"
Jul 28 15:33:17.556336 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.556246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:17.556519 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.556248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:17.556519 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:17.556372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:17.556519 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:17.556476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:17.556519 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.556248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:17.556730 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:17.556575    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:17.574524 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.574434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:17.675395 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.675361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:17.835586 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.835508    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:33:17.836014 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.835988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"b23251a34710dc34f61781120e161fe44c507849e806e9f801c6c4a80d29a15f"}
Jul 28 15:33:17.838255 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.838230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6" event={"ID":"9d95b1e3-d237-49d3-b3b0-076f7cf09367","Type":"ContainerStarted","Data":"841ababed4cbb96b1f4f3cc48cade89950ddc1b00934bd5cf92302aa10d90580"}
Jul 28 15:33:17.860824 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:17.860758    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkl6" podStartSLOduration=3.911686424 podStartE2EDuration="24.860737162s" totalImagesPullingTime="20.949050738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:17.860154558 +0000 UTC m=+25.860394484" watchObservedRunningTime="2026-07-28 15:33:17.860737162 +0000 UTC m=+25.860977095"
Jul 28 15:33:19.556446 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.556062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:19.557196 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.556071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:19.557196 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:19.556523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:19.557196 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.556077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:19.557196 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:19.556614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:19.557196 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:19.556747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:19.845373 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.845346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:33:19.845720 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.845700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"7eadf3cef0cafc002a69da57887bcc8d2ecce842a87fe49473a447f84e25c610"}
Jul 28 15:33:19.845946 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.845927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:19.846067 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.845986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:19.846274 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.846156    2577 scope.go:117] "RemoveContainer" containerID="ceb033df627b282032b9ef0a344becdfe11cc1d0251ba6b8ae7404e7df70d90c"
Jul 28 15:33:19.847703 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.847677    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="1adbfd6ee55166407ca14570131957254873f9200639c6847795f62dd65820b1" exitCode=0
Jul 28 15:33:19.847809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.847737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"1adbfd6ee55166407ca14570131957254873f9200639c6847795f62dd65820b1"}
Jul 28 15:33:19.863419 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.863391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:19.863695 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:19.863678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:20.781685 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.781649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:20.853822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.853751    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:33:20.854198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.854171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" event={"ID":"0de70cee-1194-44a8-b74a-420221243eb4","Type":"ContainerStarted","Data":"080ce792602f268b4842bfecc771ccf95b2e4008adf0c498c30b40480c4b7c1c"}
Jul 28 15:33:20.854352 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.854226    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 15:33:20.856110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.856072    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="c5c0fa4706538245c888536e9c88010bc742d0610a8dbab85bac749e8062f9eb" exitCode=0
Jul 28 15:33:20.856198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.856136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"c5c0fa4706538245c888536e9c88010bc742d0610a8dbab85bac749e8062f9eb"}
Jul 28 15:33:20.883186 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.883148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:20.893107 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:20.893056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d" podStartSLOduration=9.080894557 podStartE2EDuration="27.893041358s" totalImagesPullingTime="18.812146801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:20.891017733 +0000 UTC m=+28.891257668" watchObservedRunningTime="2026-07-28 15:33:20.893041358 +0000 UTC m=+28.893281294"
Jul 28 15:33:21.557177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.557140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:21.557329 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.557152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:21.557329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.557285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:21.557432 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.557350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:21.557432 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.557153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:21.557515 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.557431    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:21.570957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.570922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:21.672143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.671949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:21.692599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.692572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2b2bp"]
Jul 28 15:33:21.696986 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.696949    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmd7f"]
Jul 28 15:33:21.697533 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.697513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk25s"]
Jul 28 15:33:21.775243 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.775213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:21.860624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.860594    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="7f28be839906ca608c68f7d3d5696aa88a87b39760c6e8c62999762cde97a323" exitCode=0
Jul 28 15:33:21.861039 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.860685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"7f28be839906ca608c68f7d3d5696aa88a87b39760c6e8c62999762cde97a323"}
Jul 28 15:33:21.861039 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.860749    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 15:33:21.861039 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.860837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:21.861179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.861080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:21.861301 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.861226    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:21.861301 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.861278    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:21.861401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.861327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:21.861533 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:21.861400    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:21.878588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:21.878565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:22.459510 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:22.459478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:22.560406 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:22.560368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:23.442861 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.442838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:23.542020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.541986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:23.556489 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.556456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:23.556638 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.556559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:23.556638 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:23.556571    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:23.556638 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.556587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:23.556780 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:23.556674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:23.556780 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:23.556755    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:23.901706 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:23.901670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:25.556720 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:25.556686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:25.557379 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:25.556806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:25.557379 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:25.556814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:25.557379 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:25.556808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:25.557379 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:25.556922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:25.557379 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:25.557014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:26.233379 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:26.233348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:26.233533 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:26.233404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:26.233533 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.233492    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:26.233533 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.233501    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:26.233633 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.233543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret podName:019a3992-b0cd-4983-b0b6-57510bb4458c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:58.233530601 +0000 UTC m=+66.233770505 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret") pod "global-pull-secret-syncer-2b2bp" (UID: "019a3992-b0cd-4983-b0b6-57510bb4458c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:26.233633 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.233555    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:58.233549393 +0000 UTC m=+66.233789296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:26.333899 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:26.333857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:26.334066 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.334030    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:26.334066 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.334050    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:26.334066 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.334060    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zftr9 for pod openshift-network-diagnostics/network-check-target-wmd7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:26.334205 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:26.334131    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9 podName:81dbfd93-3427-4a4b-a19d-5bdb8e59e26c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:58.334116105 +0000 UTC m=+66.334356025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zftr9" (UniqueName: "kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9") pod "network-check-target-wmd7f" (UID: "81dbfd93-3427-4a4b-a19d-5bdb8e59e26c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:26.874004 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:26.873968    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="36c70c83f74e940c823d1a920840136ff274e4e2b0bdf9e82cc502a673f1aaed" exitCode=0
Jul 28 15:33:26.874593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:26.874013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"36c70c83f74e940c823d1a920840136ff274e4e2b0bdf9e82cc502a673f1aaed"}
Jul 28 15:33:27.450405 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.450371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:27.556588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.556554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:27.556833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.556597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:27.556833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.556648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:27.556833 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:27.556660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmd7f" podUID="81dbfd93-3427-4a4b-a19d-5bdb8e59e26c"
Jul 28 15:33:27.556833 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:27.556724    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk25s" podUID="b9e7bcf7-53f3-4c4b-97dd-5764f946f446"
Jul 28 15:33:27.556833 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:27.556770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2b2bp" podUID="019a3992-b0cd-4983-b0b6-57510bb4458c"
Jul 28 15:33:27.878595 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.878562    2577 generic.go:356] "Generic (PLEG): container finished" podID="eafbb7e1-e47c-429d-90df-3165977d0ae8" containerID="f77f08267f27f7196d6b4a2fe4c91b67348dd3be73b561d449baed9e66b1201a" exitCode=0
Jul 28 15:33:27.878981 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:27.878606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerDied","Data":"f77f08267f27f7196d6b4a2fe4c91b67348dd3be73b561d449baed9e66b1201a"}
Jul 28 15:33:28.294672 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.294491    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-118.ec2.internal" event="NodeReady"
Jul 28 15:33:28.294830 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.294788    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 15:33:28.354429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.354007    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m7mgz"]
Jul 28 15:33:28.377132 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.377089    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-njtkj"]
Jul 28 15:33:28.377281 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.377203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.380702 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.380572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 15:33:28.380890 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.380875    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkn7m\""
Jul 28 15:33:28.380973 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.380936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 15:33:28.397563 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.397540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7mgz"]
Jul 28 15:33:28.397681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.397588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-njtkj"]
Jul 28 15:33:28.397681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.397651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.400620 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.400601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lf5lf\""
Jul 28 15:33:28.400724 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.400607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 15:33:28.400724 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.400650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 15:33:28.400985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.400971    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 15:33:28.531035 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.531000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:28.549190 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8st\" (UniqueName: \"kubernetes.io/projected/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-kube-api-access-sh8st\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.549190 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdd9l\" (UniqueName: \"kubernetes.io/projected/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-kube-api-access-tdd9l\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.549333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-config-volume\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.549333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.549333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-tmp-dir\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.549333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.549332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-cert\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.649942 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.649899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.649942 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.649948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-tmp-dir\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.649966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-cert\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.650008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8st\" (UniqueName: \"kubernetes.io/projected/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-kube-api-access-sh8st\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.650033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdd9l\" (UniqueName: \"kubernetes.io/projected/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-kube-api-access-tdd9l\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.650062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-config-volume\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:28.650067    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:28.650188 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:28.650161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:29.150139875 +0000 UTC m=+37.150379789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:28.650415 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.650339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-tmp-dir\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.650653 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.650635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-config-volume\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.654249 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.654229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-cert\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.664307 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.664273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdd9l\" (UniqueName: \"kubernetes.io/projected/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-kube-api-access-tdd9l\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:28.664641 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.664624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8st\" (UniqueName: \"kubernetes.io/projected/bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454-kube-api-access-sh8st\") pod \"ingress-canary-njtkj\" (UID: \"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454\") " pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.704989 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.704957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-njtkj"
Jul 28 15:33:28.887368 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.887335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" event={"ID":"eafbb7e1-e47c-429d-90df-3165977d0ae8","Type":"ContainerStarted","Data":"06cd0653f5bbb5b388b55b7180515a35444d10e67bfcd0bcf7c94ce7488fedfc"}
Jul 28 15:33:28.897197 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.896893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-njtkj"]
Jul 28 15:33:28.899777 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:33:28.899752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9c2b6f_bd30_4fc7_b4cc_a94f4b543454.slice/crio-b02e2c8efef8d067246f9cfca394196c40e7b3d987bcd880e19e41e3e5682be7 WatchSource:0}: Error finding container b02e2c8efef8d067246f9cfca394196c40e7b3d987bcd880e19e41e3e5682be7: Status 404 returned error can't find the container with id b02e2c8efef8d067246f9cfca394196c40e7b3d987bcd880e19e41e3e5682be7
Jul 28 15:33:28.917167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:28.917125    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fmx6g" podStartSLOduration=7.308569695 podStartE2EDuration="35.917088867s" totalImagesPullingTime="28.608519172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.91540616 +0000 UTC m=+36.915646108" watchObservedRunningTime="2026-07-28 15:33:28.917088867 +0000 UTC m=+36.917328791"
Jul 28 15:33:29.053957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.053874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:29.142973 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.142885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:29.154341 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.154318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:29.154493 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:29.154468    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:29.154553 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:29.154533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:30.154513811 +0000 UTC m=+38.154753733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:29.556633 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.556602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:29.556872 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.556640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:29.556872 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.556640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:29.559957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.559933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:33:29.560076 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.559933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2697\""
Jul 28 15:33:29.561400 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.561350    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:33:29.561400 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.561371    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:33:29.561573 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.561492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25sjx\""
Jul 28 15:33:29.561646 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.561582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:33:29.891570 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:29.891483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-njtkj" event={"ID":"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454","Type":"ContainerStarted","Data":"b02e2c8efef8d067246f9cfca394196c40e7b3d987bcd880e19e41e3e5682be7"}
Jul 28 15:33:30.163170 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:30.163073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:30.163329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:30.163256    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:30.163329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:30.163327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:32.16330739 +0000 UTC m=+40.163547292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:31.897933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:31.897896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-njtkj" event={"ID":"bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454","Type":"ContainerStarted","Data":"15669f6a13f2d3053ff8c403551af2f01c33908c24a1971562897ffca53e9dbe"}
Jul 28 15:33:31.917665 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:31.917620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-njtkj" podStartSLOduration=1.382842278 podStartE2EDuration="3.91760666s" totalImagesPullingTime="2.534764382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:31.917413613 +0000 UTC m=+39.917653537" watchObservedRunningTime="2026-07-28 15:33:31.91760666 +0000 UTC m=+39.917846585"
Jul 28 15:33:32.178480 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:32.178389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:32.178650 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:32.178548    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:32.178650 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:32.178624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.178604907 +0000 UTC m=+44.178844810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:32.732386 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:32.732345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:32.832506 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:32.832473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:34.645640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:34.645606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:34.775278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:34.775246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:36.204039 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.203998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:36.204461 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:36.204138    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:36.204461 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:36.204188    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.204175192 +0000 UTC m=+52.204415095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:36.857855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.857823    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-599f5d8cdf-2lpp5"]
Jul 28 15:33:36.900891 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.900861    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-599f5d8cdf-2lpp5"]
Jul 28 15:33:36.901045 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.900953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:36.905030 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.905003    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 15:33:36.905170 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.905059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 15:33:36.905385 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.905365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zdkhk\""
Jul 28 15:33:36.905508 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.905365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 15:33:36.905508 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.905415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 15:33:36.906134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.906116    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 15:33:36.906219 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.906196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 15:33:36.908160 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.908139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn689\" (UniqueName: \"kubernetes.io/projected/402e1330-0b95-401f-8dbf-17c90f19df84-kube-api-access-xn689\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:36.908258 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.908174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-stats-auth\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:36.908258 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.908217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-default-certificate\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:36.908258 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.908241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:36.908419 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:36.908335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.008983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.008951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn689\" (UniqueName: \"kubernetes.io/projected/402e1330-0b95-401f-8dbf-17c90f19df84-kube-api-access-xn689\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.009177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.008989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-stats-auth\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.009177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.009036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-default-certificate\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.009177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.009062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.009177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.009131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.009329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.009211    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:37.009329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.009291    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.509274025 +0000 UTC m=+45.509513928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:37.009329 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.009311    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.509302751 +0000 UTC m=+45.509542654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:37.012588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.012565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-default-certificate\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.012681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.012631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-stats-auth\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.022204 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.022181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn689\" (UniqueName: \"kubernetes.io/projected/402e1330-0b95-401f-8dbf-17c90f19df84-kube-api-access-xn689\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.034500 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.034464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:37.111164 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.111070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:37.512783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.512743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.513252 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:37.512805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:37.513252 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.512921    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:37.513252 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.512928    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.512913707 +0000 UTC m=+46.513153610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:37.513252 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:37.513009    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.512990773 +0000 UTC m=+46.513230681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:38.520072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.520036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:38.520526 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.520123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:38.520526 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:38.520204    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:38.520526 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:38.520268    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:40.520242374 +0000 UTC m=+48.520482291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:38.520526 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:38.520301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:40.520290848 +0000 UTC m=+48.520530757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:38.532628 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.532597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.636318 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.636290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.877026 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.876941    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.961321 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:38.961275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:40.532992 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:40.532954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:40.533444 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:40.533004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:40.533444 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:40.533171    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:40.533444 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:40.533178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.53316192 +0000 UTC m=+52.533401837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:40.533444 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:40.533259    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.533241249 +0000 UTC m=+52.533481156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:41.722799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:41.722768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:41.799505 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:41.799475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:43.706712 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.706680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:43.787609 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.787576    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8ppfv"]
Jul 28 15:33:43.802968 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.802946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.805327 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.805302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ppfv"]
Jul 28 15:33:43.806678 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.806657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 15:33:43.807975 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.807951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgs6z\""
Jul 28 15:33:43.807975 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.807973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 15:33:43.808178 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.808135    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 15:33:43.808178 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.808172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 15:33:43.815393 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.815372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:43.857374 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.857342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vwc\" (UniqueName: \"kubernetes.io/projected/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-api-access-m2vwc\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.857503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.857428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32fbe8d-6805-444c-a3e5-261b6f6e828f-crio-socket\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.857503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.857454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.857503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.857486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.857603 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.857521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32fbe8d-6805-444c-a3e5-261b6f6e828f-data-volume\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.905799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.905767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:43.957826 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.957757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32fbe8d-6805-444c-a3e5-261b6f6e828f-crio-socket\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.957826 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.957786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.957826 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.957823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:43.957887    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:43.957907    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:43.957949    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.457934227 +0000 UTC m=+52.458174129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.957967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32fbe8d-6805-444c-a3e5-261b6f6e828f-data-volume\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.958014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vwc\" (UniqueName: \"kubernetes.io/projected/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-api-access-m2vwc\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.958013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32fbe8d-6805-444c-a3e5-261b6f6e828f-crio-socket\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.958057 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:43.958036    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.458020401 +0000 UTC m=+52.458260305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:43.958346 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.958328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32fbe8d-6805-444c-a3e5-261b6f6e828f-data-volume\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:43.969712 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:43.969683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vwc\" (UniqueName: \"kubernetes.io/projected/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-api-access-m2vwc\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:44.260578 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:44.260547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:33:44.260730 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.260704    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:44.260776 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.260766    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls podName:02dda83b-0c24-4d24-9aa0-5d7da07ca16b nodeName:}" failed. No retries permitted until 2026-07-28 15:34:00.260749573 +0000 UTC m=+68.260989496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls") pod "dns-default-m7mgz" (UID: "02dda83b-0c24-4d24-9aa0-5d7da07ca16b") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:44.462695 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:44.462659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:44.462842 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:44.462710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:44.462842 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.462780    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:44.462842 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.462808    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:44.462842 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.462840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.462825699 +0000 UTC m=+53.463065602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:44.462976 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.462853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.462847469 +0000 UTC m=+53.463087373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:44.563606 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:44.563528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:44.563606 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:44.563581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:44.563803 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.563689    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:44.563803 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.563734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:52.563716931 +0000 UTC m=+60.563956853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:44.563803 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:44.563752    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:52.563744699 +0000 UTC m=+60.563984601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:45.469290 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:45.469231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:45.469290 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:45.469305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:45.469788 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:45.469348    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:45.469788 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:45.469397    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:45.469788 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:45.469414    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:47.469398173 +0000 UTC m=+55.469638076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:45.469788 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:45.469436    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:47.469425048 +0000 UTC m=+55.469664951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:45.568943 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:45.568916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-df56s_cdbd0202-abf5-4432-8dc4-1197b601c3e6/dns-node-resolver/0.log"
Jul 28 15:33:46.764875 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:46.764845    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hgcb_a03cfd2f-05a8-4157-87c4-8dc38753fede/node-ca/0.log"
Jul 28 15:33:47.363680 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.363640    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-njtkj_bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454/serve-healthcheck-canary/0.log"
Jul 28 15:33:47.486394 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.486359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:47.486573 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.486421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:47.486573 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.486480    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:47.486573 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.486550    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.486531902 +0000 UTC m=+59.486771813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:47.486678 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.486585    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:47.486678 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.486648    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.48663444 +0000 UTC m=+59.486874342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:47.498531 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.498496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"]
Jul 28 15:33:47.500306 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.500283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"
Jul 28 15:33:47.503218 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.503200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bj5zr\""
Jul 28 15:33:47.517764 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.517734    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"]
Jul 28 15:33:47.586868 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.586836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9fp\" (UniqueName: \"kubernetes.io/projected/e1b0474d-60f9-4016-9ed0-51dbc9bd1f66-kube-api-access-dq9fp\") pod \"network-check-source-7d88cb55d8-czbqw\" (UID: \"e1b0474d-60f9-4016-9ed0-51dbc9bd1f66\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"
Jul 28 15:33:47.688203 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.688121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9fp\" (UniqueName: \"kubernetes.io/projected/e1b0474d-60f9-4016-9ed0-51dbc9bd1f66-kube-api-access-dq9fp\") pod \"network-check-source-7d88cb55d8-czbqw\" (UID: \"e1b0474d-60f9-4016-9ed0-51dbc9bd1f66\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"
Jul 28 15:33:47.701833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.701807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9fp\" (UniqueName: \"kubernetes.io/projected/e1b0474d-60f9-4016-9ed0-51dbc9bd1f66-kube-api-access-dq9fp\") pod \"network-check-source-7d88cb55d8-czbqw\" (UID: \"e1b0474d-60f9-4016-9ed0-51dbc9bd1f66\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"
Jul 28 15:33:47.809324 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.809282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"
Jul 28 15:33:47.824703 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.824675    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:33:47.826465 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.826449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.829506 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.829481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 15:33:47.829626 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.829523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 15:33:47.829973 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.829953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 15:33:47.830069 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.829954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xl2f2\""
Jul 28 15:33:47.835802 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.835779    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 15:33:47.839839 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.839817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:33:47.890157 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890284 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466v5\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890284 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890364 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890412 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890480 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890585 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.890629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.890597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.933239 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.933129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw"]
Jul 28 15:33:47.935752 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:33:47.935727    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b0474d_60f9_4016_9ed0_51dbc9bd1f66.slice/crio-87f8a68138f3ff7fdbd494355c357af82d94d23a26f3b49c1a7c73ace1411933 WatchSource:0}: Error finding container 87f8a68138f3ff7fdbd494355c357af82d94d23a26f3b49c1a7c73ace1411933: Status 404 returned error can't find the container with id 87f8a68138f3ff7fdbd494355c357af82d94d23a26f3b49c1a7c73ace1411933
Jul 28 15:33:47.940087 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.940062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw" event={"ID":"e1b0474d-60f9-4016-9ed0-51dbc9bd1f66","Type":"ContainerStarted","Data":"87f8a68138f3ff7fdbd494355c357af82d94d23a26f3b49c1a7c73ace1411933"}
Jul 28 15:33:47.991429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991570 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991570 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991570 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.991560    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.991585    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b778d4587-kcswv: secret "image-registry-tls" not found
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:47.991714    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls podName:9d0bf632-720d-4ada-bda7-1a2c61ac7f11 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:48.491699112 +0000 UTC m=+56.491939014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls") pod "image-registry-6b778d4587-kcswv" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11") : secret "image-registry-tls" not found
Jul 28 15:33:47.991761 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.992128 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.991778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466v5\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.992228 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.992201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.992528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.992505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.992994 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.992974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.993985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.993965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:47.994060 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:47.994035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:48.005747 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:48.005716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:48.005844 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:48.005785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-466v5\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:48.496681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:48.496636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:48.496879 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:48.496806    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:48.496879 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:48.496827    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b778d4587-kcswv: secret "image-registry-tls" not found
Jul 28 15:33:48.497064 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:48.496899    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls podName:9d0bf632-720d-4ada-bda7-1a2c61ac7f11 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:49.496875604 +0000 UTC m=+57.497115507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls") pod "image-registry-6b778d4587-kcswv" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11") : secret "image-registry-tls" not found
Jul 28 15:33:49.332948 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:49.332916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.407957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:49.407913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.481158 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:49.481124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.505452 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:49.505420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:49.505623 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:49.505549    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:49.505623 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:49.505564    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b778d4587-kcswv: secret "image-registry-tls" not found
Jul 28 15:33:49.505723 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:49.505628    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls podName:9d0bf632-720d-4ada-bda7-1a2c61ac7f11 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.505610815 +0000 UTC m=+59.505850721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls") pod "image-registry-6b778d4587-kcswv" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11") : secret "image-registry-tls" not found
Jul 28 15:33:50.941940 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:50.941903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.035527 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.035496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.378278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.378248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.478660 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.478620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.525707 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.525676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:51.525794 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.525751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:51.525794 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.525788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:51.525869 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525848    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:51.525908 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525866    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:51.525908 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525885    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:51.525962 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525869    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b778d4587-kcswv: secret "image-registry-tls" not found
Jul 28 15:33:51.525962 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525928    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:59.525913065 +0000 UTC m=+67.526152968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:51.525962 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525942    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:59.525936063 +0000 UTC m=+67.526175966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:51.525962 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:51.525953    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls podName:9d0bf632-720d-4ada-bda7-1a2c61ac7f11 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:55.52594781 +0000 UTC m=+63.526187713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls") pod "image-registry-6b778d4587-kcswv" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11") : secret "image-registry-tls" not found
Jul 28 15:33:51.659520 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.659439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.761920 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.761885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.951235 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.951147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw" event={"ID":"e1b0474d-60f9-4016-9ed0-51dbc9bd1f66","Type":"ContainerStarted","Data":"ed2a17017b284e2fcabe19ee243c50879de4e4d861fcc69683da99cc46cbac27"}
Jul 28 15:33:51.970558 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:51.970503    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-czbqw" podStartSLOduration=1.929963211 podStartE2EDuration="4.970485866s" totalImagesPullingTime="3.040522655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:51.969898709 +0000 UTC m=+59.970138656" watchObservedRunningTime="2026-07-28 15:33:51.970485866 +0000 UTC m=+59.970725791"
Jul 28 15:33:52.531322 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:52.531290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:52.634704 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:52.634658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:52.634885 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:52.634721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:33:52.634885 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:52.634808    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 15:33:52.634885 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:52.634849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.634834465 +0000 UTC m=+76.635074367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : configmap references non-existent config key: service-ca.crt
Jul 28 15:33:52.634885 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:52.634864    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs podName:402e1330-0b95-401f-8dbf-17c90f19df84 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.634857769 +0000 UTC m=+76.635097672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs") pod "router-default-599f5d8cdf-2lpp5" (UID: "402e1330-0b95-401f-8dbf-17c90f19df84") : secret "router-metrics-certs-default" not found
Jul 28 15:33:53.320937 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:53.320892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:53.397933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:53.397867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:53.911838 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:53.911812    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcw4d"
Jul 28 15:33:54.365205 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:54.365174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:55.557457 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:55.557402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:33:55.557860 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:55.557547    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:55.557860 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:55.557566    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b778d4587-kcswv: secret "image-registry-tls" not found
Jul 28 15:33:55.557860 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:55.557616    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls podName:9d0bf632-720d-4ada-bda7-1a2c61ac7f11 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:03.557602733 +0000 UTC m=+71.557842636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls") pod "image-registry-6b778d4587-kcswv" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11") : secret "image-registry-tls" not found
Jul 28 15:33:56.739730 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:56.739692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:56.921530 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:56.921454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.016150 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.016033    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.102415 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.102385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.244561 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.244527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.333809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.333727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.432850 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.432820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.961899 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:57.961863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:58.063046 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.063009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:58.278134 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.278107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:58.278306 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.278162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:33:58.284250 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.284221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:33:58.284389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.284255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:33:58.288593 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:58.288577    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 15:33:58.288660 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:58.288649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs podName:b9e7bcf7-53f3-4c4b-97dd-5764f946f446 nodeName:}" failed. No retries permitted until 2026-07-28 15:35:02.288633815 +0000 UTC m=+130.288873719 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs") pod "network-metrics-daemon-tk25s" (UID: "b9e7bcf7-53f3-4c4b-97dd-5764f946f446") : secret "metrics-daemon-secret" not found
Jul 28 15:33:58.290725 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.290704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/019a3992-b0cd-4983-b0b6-57510bb4458c-original-pull-secret\") pod \"global-pull-secret-syncer-2b2bp\" (UID: \"019a3992-b0cd-4983-b0b6-57510bb4458c\") " pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:58.369328 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.369288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2b2bp"
Jul 28 15:33:58.379406 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.379376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:58.382210 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.382179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftr9\" (UniqueName: \"kubernetes.io/projected/81dbfd93-3427-4a4b-a19d-5bdb8e59e26c-kube-api-access-zftr9\") pod \"network-check-target-wmd7f\" (UID: \"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c\") " pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:58.488855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.488819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2b2bp"]
Jul 28 15:33:58.666877 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.666792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:58.680105 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.680076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2697\""
Jul 28 15:33:58.687363 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.687347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:58.806509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.806408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmd7f"]
Jul 28 15:33:58.808863 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:33:58.808840    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dbfd93_3427_4a4b_a19d_5bdb8e59e26c.slice/crio-cc5b042649a0d2b79ad26540fb040027e0bc9dbf5988e06aa45f4ef6fa43f037 WatchSource:0}: Error finding container cc5b042649a0d2b79ad26540fb040027e0bc9dbf5988e06aa45f4ef6fa43f037: Status 404 returned error can't find the container with id cc5b042649a0d2b79ad26540fb040027e0bc9dbf5988e06aa45f4ef6fa43f037
Jul 28 15:33:58.968273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.968188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2b2bp" event={"ID":"019a3992-b0cd-4983-b0b6-57510bb4458c","Type":"ContainerStarted","Data":"c90de7c693652ef4d489311321da098a375515a03618d09e571e3ecd5a70f8f1"}
Jul 28 15:33:58.969855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.969804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmd7f" event={"ID":"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c","Type":"ContainerStarted","Data":"a58a39d3c04a1145380dd63c690fbd9114a870c49734d527934c2b3271701c86"}
Jul 28 15:33:58.969855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.969854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmd7f" event={"ID":"81dbfd93-3427-4a4b-a19d-5bdb8e59e26c","Type":"ContainerStarted","Data":"cc5b042649a0d2b79ad26540fb040027e0bc9dbf5988e06aa45f4ef6fa43f037"}
Jul 28 15:33:58.970065 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.969956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:33:58.988417 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:58.988355    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wmd7f" podStartSLOduration=65.988333811 podStartE2EDuration="1m5.988333811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:58.987408533 +0000 UTC m=+66.987648461" watchObservedRunningTime="2026-07-28 15:33:58.988333811 +0000 UTC m=+66.988573736"
Jul 28 15:33:59.031579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.031546    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:59.476352 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.476319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:59.590244 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.590208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:59.590432 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.590267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:59.590432 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:59.590421    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:59.590559 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:33:59.590547    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls podName:b32fbe8d-6805-444c-a3e5-261b6f6e828f nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.590526733 +0000 UTC m=+83.590766637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ppfv" (UID: "b32fbe8d-6805-444c-a3e5-261b6f6e828f") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:59.590865 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.590834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32fbe8d-6805-444c-a3e5-261b6f6e828f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:33:59.823292 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:33:59.823149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:00.027692 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.027662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:00.296638 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.296602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:34:00.299274 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.299254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dda83b-0c24-4d24-9aa0-5d7da07ca16b-metrics-tls\") pod \"dns-default-m7mgz\" (UID: \"02dda83b-0c24-4d24-9aa0-5d7da07ca16b\") " pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:34:00.408318 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.408281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:00.489530 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.489501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkn7m\""
Jul 28 15:34:00.497366 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.497337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:34:00.511424 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.511398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:00.638900 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.638873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7mgz"]
Jul 28 15:34:00.641898 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:00.641866    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dda83b_0c24_4d24_9aa0_5d7da07ca16b.slice/crio-5fd1ba8bf074f9a18ce6698788f2d50260786a3cad0bea027f091357107144d0 WatchSource:0}: Error finding container 5fd1ba8bf074f9a18ce6698788f2d50260786a3cad0bea027f091357107144d0: Status 404 returned error can't find the container with id 5fd1ba8bf074f9a18ce6698788f2d50260786a3cad0bea027f091357107144d0
Jul 28 15:34:00.837679 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.837596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:00.977011 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:00.976975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mgz" event={"ID":"02dda83b-0c24-4d24-9aa0-5d7da07ca16b","Type":"ContainerStarted","Data":"5fd1ba8bf074f9a18ce6698788f2d50260786a3cad0bea027f091357107144d0"}
Jul 28 15:34:01.019481 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:01.019439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:01.423906 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:01.423854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:01.827089 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:01.827050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.031224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:02.031191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.424744 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:02.424655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.984443 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:02.984397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2b2bp" event={"ID":"019a3992-b0cd-4983-b0b6-57510bb4458c","Type":"ContainerStarted","Data":"a6f4faa85f2cc18c69ffdd6d717b83123dcbd62262841ebc3349cb4a3bebf2c0"}
Jul 28 15:34:03.002066 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.002006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2b2bp" podStartSLOduration=66.450005007 podStartE2EDuration="1m10.001985454s" totalImagesPullingTime="3.551980447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:03.000951299 +0000 UTC m=+71.001191261" watchObservedRunningTime="2026-07-28 15:34:03.001985454 +0000 UTC m=+71.002225381"
Jul 28 15:34:03.023594 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.023562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.090945 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.090906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.618635 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.618552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.627657 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.627630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:03.629995 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.629971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"image-registry-6b778d4587-kcswv\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") " pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:03.738989 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.738952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xl2f2\""
Jul 28 15:34:03.745697 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.745681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:03.833854 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.833820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.877423 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.877344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:34:03.880881 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:03.880856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0bf632_720d_4ada_bda7_1a2c61ac7f11.slice/crio-35c303dae8478935930c6f92e7bde4fa8051e3d42e9acb4b731635f9bb3d8b1c WatchSource:0}: Error finding container 35c303dae8478935930c6f92e7bde4fa8051e3d42e9acb4b731635f9bb3d8b1c: Status 404 returned error can't find the container with id 35c303dae8478935930c6f92e7bde4fa8051e3d42e9acb4b731635f9bb3d8b1c
Jul 28 15:34:03.989295 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.989255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" event={"ID":"9d0bf632-720d-4ada-bda7-1a2c61ac7f11","Type":"ContainerStarted","Data":"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"}
Jul 28 15:34:03.989486 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.989325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:03.989486 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.989350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" event={"ID":"9d0bf632-720d-4ada-bda7-1a2c61ac7f11","Type":"ContainerStarted","Data":"35c303dae8478935930c6f92e7bde4fa8051e3d42e9acb4b731635f9bb3d8b1c"}
Jul 28 15:34:03.991068 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.991038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mgz" event={"ID":"02dda83b-0c24-4d24-9aa0-5d7da07ca16b","Type":"ContainerStarted","Data":"1afc7f8b74ba2fa8037eabb01b1a0e22bc9584c968dd72cbdcb4e3c2862a671a"}
Jul 28 15:34:03.991204 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.991115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mgz" event={"ID":"02dda83b-0c24-4d24-9aa0-5d7da07ca16b","Type":"ContainerStarted","Data":"450e152abd58fd22f3241461c49c69765d88324d175780c0ac134695b4b18e1c"}
Jul 28 15:34:03.991253 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:03.991230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:34:04.012428 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:04.012379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" podStartSLOduration=17.012363882 podStartE2EDuration="17.012363882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:04.011660297 +0000 UTC m=+72.011900235" watchObservedRunningTime="2026-07-28 15:34:04.012363882 +0000 UTC m=+72.012603808"
Jul 28 15:34:04.031462 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:04.031407    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m7mgz" podStartSLOduration=33.450110404 podStartE2EDuration="36.031393284s" totalImagesPullingTime="2.58128288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:04.030468975 +0000 UTC m=+72.030708913" watchObservedRunningTime="2026-07-28 15:34:04.031393284 +0000 UTC m=+72.031633208"
Jul 28 15:34:04.218016 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:04.217930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:04.421237 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:04.421204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.016300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:05.016268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.447198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:05.447117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.619355 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:05.619324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.783618 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:05.783586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.849892 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:05.849866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:06.175273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:06.175193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:06.274795 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:06.274767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:06.819847 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:06.819811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.022766 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:07.022736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.221450 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:07.221376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.440221 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:07.440192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:08.670981 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.670928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:08.670981 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.670990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:08.671648 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.671627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e1330-0b95-401f-8dbf-17c90f19df84-service-ca-bundle\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:08.673306 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.673289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402e1330-0b95-401f-8dbf-17c90f19df84-metrics-certs\") pod \"router-default-599f5d8cdf-2lpp5\" (UID: \"402e1330-0b95-401f-8dbf-17c90f19df84\") " pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:08.713162 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.713128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zdkhk\""
Jul 28 15:34:08.720637 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.720617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:08.851127 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:08.851082    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-599f5d8cdf-2lpp5"]
Jul 28 15:34:08.854482 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:08.854440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402e1330_0b95_401f_8dbf_17c90f19df84.slice/crio-4a68df3ba3096b695b04aa9f7e101c0db29926662f353a3ea9cc6238bc37ce41 WatchSource:0}: Error finding container 4a68df3ba3096b695b04aa9f7e101c0db29926662f353a3ea9cc6238bc37ce41: Status 404 returned error can't find the container with id 4a68df3ba3096b695b04aa9f7e101c0db29926662f353a3ea9cc6238bc37ce41
Jul 28 15:34:09.011867 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:09.011827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5" event={"ID":"402e1330-0b95-401f-8dbf-17c90f19df84","Type":"ContainerStarted","Data":"4ac2c6540f7907ae5c5ee66a3bdd91f24cd085be1a394a9c282eb4daa9ade6ba"}
Jul 28 15:34:09.012056 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:09.011873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5" event={"ID":"402e1330-0b95-401f-8dbf-17c90f19df84","Type":"ContainerStarted","Data":"4a68df3ba3096b695b04aa9f7e101c0db29926662f353a3ea9cc6238bc37ce41"}
Jul 28 15:34:09.036417 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:09.036362    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5" podStartSLOduration=33.036341084 podStartE2EDuration="33.036341084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:09.035579122 +0000 UTC m=+77.035819055" watchObservedRunningTime="2026-07-28 15:34:09.036341084 +0000 UTC m=+77.036581009"
Jul 28 15:34:09.721456 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:09.721412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:09.724154 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:09.724132    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:10.015967 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:10.015940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:10.017687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:10.017650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-599f5d8cdf-2lpp5"
Jul 28 15:34:12.390772 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:12.390732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.495983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:12.495947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.593374 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:12.593342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.690521 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:12.690442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:14.000304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.000276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m7mgz"
Jul 28 15:34:14.489806 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.489775    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"]
Jul 28 15:34:14.492745 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.492728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.518082 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.518055    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 15:34:14.519506 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519483    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 15:34:14.519506 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 15:34:14.519588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 15:34:14.519588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 15:34:14.519661 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 15:34:14.519661 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.519483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 15:34:14.571552 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.571518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"]
Jul 28 15:34:14.616459 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.616619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bns\" (UniqueName: \"kubernetes.io/projected/9bc1f4ac-039c-41d5-b153-c71e5d370171-kube-api-access-f6bns\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.616619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.616619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1f4ac-039c-41d5-b153-c71e5d370171-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.616726 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.616726 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.616662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.676179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.676141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:34:14.717262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.717262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bns\" (UniqueName: \"kubernetes.io/projected/9bc1f4ac-039c-41d5-b153-c71e5d370171-kube-api-access-f6bns\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.717442 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.717442 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1f4ac-039c-41d5-b153-c71e5d370171-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.717508 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.717551 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.717518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.718056 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.718031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc1f4ac-039c-41d5-b153-c71e5d370171-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.719833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.719804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.719833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.719822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.720088 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.720067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-ca\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.720148 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.720130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bc1f4ac-039c-41d5-b153-c71e5d370171-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.747285 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.747223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bns\" (UniqueName: \"kubernetes.io/projected/9bc1f4ac-039c-41d5-b153-c71e5d370171-kube-api-access-f6bns\") pod \"cluster-proxy-proxy-agent-85b86bddd9-lmrd6\" (UID: \"9bc1f4ac-039c-41d5-b153-c71e5d370171\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.770206 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.770176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:14.816217 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.816185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"
Jul 28 15:34:14.852332 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.852305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:14.941832 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.941803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:14.959026 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:14.958999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6"]
Jul 28 15:34:14.962027 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:14.962002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bc1f4ac_039c_41d5_b153_c71e5d370171.slice/crio-00203ee9d5e3a504ec5439d2a9d77cb0ee7525dc788028682a79531fe7696932 WatchSource:0}: Error finding container 00203ee9d5e3a504ec5439d2a9d77cb0ee7525dc788028682a79531fe7696932: Status 404 returned error can't find the container with id 00203ee9d5e3a504ec5439d2a9d77cb0ee7525dc788028682a79531fe7696932
Jul 28 15:34:15.031088 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.031003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:15.032855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.032829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6" event={"ID":"9bc1f4ac-039c-41d5-b153-c71e5d370171","Type":"ContainerStarted","Data":"00203ee9d5e3a504ec5439d2a9d77cb0ee7525dc788028682a79531fe7696932"}
Jul 28 15:34:15.626647 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.626612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:34:15.629503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.629472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32fbe8d-6805-444c-a3e5-261b6f6e828f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ppfv\" (UID: \"b32fbe8d-6805-444c-a3e5-261b6f6e828f\") " pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:34:15.917230 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.917147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgs6z\""
Jul 28 15:34:15.924187 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:15.924160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ppfv"
Jul 28 15:34:16.083493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:16.083397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ppfv"]
Jul 28 15:34:16.088044 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:16.088015    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32fbe8d_6805_444c_a3e5_261b6f6e828f.slice/crio-fe2b268dd2893b5b463aa53b6cc2d32069b5061079b0e15780f253528c58ab1a WatchSource:0}: Error finding container fe2b268dd2893b5b463aa53b6cc2d32069b5061079b0e15780f253528c58ab1a: Status 404 returned error can't find the container with id fe2b268dd2893b5b463aa53b6cc2d32069b5061079b0e15780f253528c58ab1a
Jul 28 15:34:17.041774 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:17.041735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ppfv" event={"ID":"b32fbe8d-6805-444c-a3e5-261b6f6e828f","Type":"ContainerStarted","Data":"fe2b268dd2893b5b463aa53b6cc2d32069b5061079b0e15780f253528c58ab1a"}
Jul 28 15:34:17.915401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:17.915285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.985738 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:17.985702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.640162 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:18.640060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.733303 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:18.733269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.058397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:19.058363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6" event={"ID":"9bc1f4ac-039c-41d5-b153-c71e5d370171","Type":"ContainerStarted","Data":"33777fd1096030ac17195b384ddecda0ccba0773d099150f483a46f4314b2e87"}
Jul 28 15:34:19.058770 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:19.058412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6" event={"ID":"9bc1f4ac-039c-41d5-b153-c71e5d370171","Type":"ContainerStarted","Data":"3cbbe1e7d519f0926da6b2d7fb389dc3e03355046bcdfde51067a6f38b551330"}
Jul 28 15:34:19.058770 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:19.058430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6" event={"ID":"9bc1f4ac-039c-41d5-b153-c71e5d370171","Type":"ContainerStarted","Data":"5ce471b111a37d200ced3fd4cf72546bfb017b6839066785cb6fdbdf086298fb"}
Jul 28 15:34:19.066113 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:19.066075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.080963 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:19.080916    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b86bddd9-lmrd6" podStartSLOduration=1.4012905199999999 podStartE2EDuration="5.080900202s" totalImagesPullingTime="3.679609682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:19.07883894 +0000 UTC m=+87.079078868" watchObservedRunningTime="2026-07-28 15:34:19.080900202 +0000 UTC m=+87.081140127"
Jul 28 15:34:20.786171 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:20.786136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:20.855282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:20.855247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.070272 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:22.070237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ppfv" event={"ID":"b32fbe8d-6805-444c-a3e5-261b6f6e828f","Type":"ContainerStarted","Data":"80045f49f5fd44619d1498291fc1e47ca106c136b13e2f996019b7b1a056e2ca"}
Jul 28 15:34:23.576167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:23.576124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:23.680378 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:23.680345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:23.862437 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:23.862358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.682292 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:24.682258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:25.576830 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:25.576794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.674070 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:25.673983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.776815 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:25.776780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.875347 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:25.875308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.977793 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:25.977761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.086933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.086896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ppfv" event={"ID":"b32fbe8d-6805-444c-a3e5-261b6f6e828f","Type":"ContainerStarted","Data":"95d3d9c546cc3c31f6ca8d8e7be13181ee9425b5759a35a8e18c6fff100aede2"}
Jul 28 15:34:26.087907 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.087874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.090897 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.090874    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lrs8d"]
Jul 28 15:34:26.093226 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.093121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.096080 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.096057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4lhqk\""
Jul 28 15:34:26.096222 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.096057    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 15:34:26.097192 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.097170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 15:34:26.097297 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.097269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 15:34:26.097297 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.097273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 15:34:26.097414 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.097317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 15:34:26.102484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.102464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 15:34:26.194404 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.194371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.214858 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.214826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9x4\" (UniqueName: \"kubernetes.io/projected/1028cb4b-8c23-4a1d-875d-871e134b7358-kube-api-access-8g9x4\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215028 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.214871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215028 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.214914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-metrics-client-ca\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215028 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.214939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-sys\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215028 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.214984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-root\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215028 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.215021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.215122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-textfile\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.215157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-tls\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.215447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.215180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-wtmp\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.295282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.295194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.316196 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-textfile\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-tls\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-wtmp\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9x4\" (UniqueName: \"kubernetes.io/projected/1028cb4b-8c23-4a1d-875d-871e134b7358-kube-api-access-8g9x4\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-metrics-client-ca\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-sys\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-root\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-wtmp\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-root\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1028cb4b-8c23-4a1d-875d-871e134b7358-sys\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.316676 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.316598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-textfile\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.317160 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.317140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-metrics-client-ca\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.317363 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.317325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.319420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.319403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-tls\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.319509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.319443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1028cb4b-8c23-4a1d-875d-871e134b7358-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.326630 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.326608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9x4\" (UniqueName: \"kubernetes.io/projected/1028cb4b-8c23-4a1d-875d-871e134b7358-kube-api-access-8g9x4\") pod \"node-exporter-lrs8d\" (UID: \"1028cb4b-8c23-4a1d-875d-871e134b7358\") " pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.399021 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.398987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.405003 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.404976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrs8d"
Jul 28 15:34:26.416750 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:26.416720    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1028cb4b_8c23_4a1d_875d_871e134b7358.slice/crio-673c24e35ae218e1dbbde79105dd34853afe99263f248255ab0ed52579d05802 WatchSource:0}: Error finding container 673c24e35ae218e1dbbde79105dd34853afe99263f248255ab0ed52579d05802: Status 404 returned error can't find the container with id 673c24e35ae218e1dbbde79105dd34853afe99263f248255ab0ed52579d05802
Jul 28 15:34:26.495700 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.495665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.796649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:26.796613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.080298 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.080221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.092194 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.092156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrs8d" event={"ID":"1028cb4b-8c23-4a1d-875d-871e134b7358","Type":"ContainerStarted","Data":"673c24e35ae218e1dbbde79105dd34853afe99263f248255ab0ed52579d05802"}
Jul 28 15:34:27.160997 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.160962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.279866 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.279829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.507064 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.507030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.675201 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.675164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.983412 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:27.983341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.084734 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:28.084702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.221009 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:28.220962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.001024 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.000998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.093024 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.092992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.100598 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.100557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ppfv" event={"ID":"b32fbe8d-6805-444c-a3e5-261b6f6e828f","Type":"ContainerStarted","Data":"e9c922ec32d45b900f3b57809a2cdc4dab90d773cd72f83a5aa38032e18bc2b6"}
Jul 28 15:34:29.102120 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.102052    2577 generic.go:356] "Generic (PLEG): container finished" podID="1028cb4b-8c23-4a1d-875d-871e134b7358" containerID="81b3df273632f3a7592cb30b47d92e3734b2d85ca2ccf80453f707e018056a91" exitCode=0
Jul 28 15:34:29.102237 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.102134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrs8d" event={"ID":"1028cb4b-8c23-4a1d-875d-871e134b7358","Type":"ContainerDied","Data":"81b3df273632f3a7592cb30b47d92e3734b2d85ca2ccf80453f707e018056a91"}
Jul 28 15:34:29.119905 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.119863    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8ppfv" podStartSLOduration=33.367721621 podStartE2EDuration="46.119848624s" totalImagesPullingTime="12.752127003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:29.119298683 +0000 UTC m=+97.119538625" watchObservedRunningTime="2026-07-28 15:34:29.119848624 +0000 UTC m=+97.120088548"
Jul 28 15:34:29.165510 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.165486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.353485 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.353452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.975438 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:29.975409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wmd7f"
Jul 28 15:34:30.107728 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.107686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrs8d" event={"ID":"1028cb4b-8c23-4a1d-875d-871e134b7358","Type":"ContainerStarted","Data":"2427673c90d821812618675bf804573afeb866dfff0cc78e36b0fcfbae22cdab"}
Jul 28 15:34:30.108137 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.107754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrs8d" event={"ID":"1028cb4b-8c23-4a1d-875d-871e134b7358","Type":"ContainerStarted","Data":"99f3bdb56607f30cab3c373c6df90ee5a578fc16ee9fccebc9ebef93b4807947"}
Jul 28 15:34:30.126498 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.126447    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lrs8d" podStartSLOduration=1.5389764769999998 podStartE2EDuration="4.126432304s" totalImagesPullingTime="2.524373523s" totalInitContainerRuntime="63.082304ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:30.12571058 +0000 UTC m=+98.125950505" watchObservedRunningTime="2026-07-28 15:34:30.126432304 +0000 UTC m=+98.126672229"
Jul 28 15:34:30.839716 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.839679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.848803 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.848780    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"]
Jul 28 15:34:30.853017 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.853001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:30.855620 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.855598    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 15:34:30.855735 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.855689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c2dpp\""
Jul 28 15:34:30.861159 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.861138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"]
Jul 28 15:34:30.919747 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.919717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.954871 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.954843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:30.955002 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:30.954901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcgw\" (UniqueName: \"kubernetes.io/projected/4f37889f-e173-4483-ad9a-38e455c2954d-kube-api-access-rtcgw\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.056173 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.056142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.056340 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.056196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcgw\" (UniqueName: \"kubernetes.io/projected/4f37889f-e173-4483-ad9a-38e455c2954d-kube-api-access-rtcgw\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.056340 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:34:31.056301    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 15:34:31.056472 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:34:31.056377    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert podName:4f37889f-e173-4483-ad9a-38e455c2954d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:31.55635613 +0000 UTC m=+99.556596033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-v5pwb" (UID: "4f37889f-e173-4483-ad9a-38e455c2954d") : secret "monitoring-plugin-cert" not found
Jul 28 15:34:31.066222 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.066203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcgw\" (UniqueName: \"kubernetes.io/projected/4f37889f-e173-4483-ad9a-38e455c2954d-kube-api-access-rtcgw\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.561495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.561459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.563853 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.563829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f37889f-e173-4483-ad9a-38e455c2954d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v5pwb\" (UID: \"4f37889f-e173-4483-ad9a-38e455c2954d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.570240 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.570209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:31.762839 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.762806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:31.887797 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:31.887763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"]
Jul 28 15:34:31.890697 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:31.890674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f37889f_e173_4483_ad9a_38e455c2954d.slice/crio-412c7bf6aa287d12a3ee83a9808f93fd02a38fc12c20b9764aab358fd4d27ca2 WatchSource:0}: Error finding container 412c7bf6aa287d12a3ee83a9808f93fd02a38fc12c20b9764aab358fd4d27ca2: Status 404 returned error can't find the container with id 412c7bf6aa287d12a3ee83a9808f93fd02a38fc12c20b9764aab358fd4d27ca2
Jul 28 15:34:32.115160 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.115054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb" event={"ID":"4f37889f-e173-4483-ad9a-38e455c2954d","Type":"ContainerStarted","Data":"412c7bf6aa287d12a3ee83a9808f93fd02a38fc12c20b9764aab358fd4d27ca2"}
Jul 28 15:34:32.275586 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.275551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.388386 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.388298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.595023 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.594988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.898899 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.898862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.983761 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:32.983728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.123257 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.123224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb" event={"ID":"4f37889f-e173-4483-ad9a-38e455c2954d","Type":"ContainerStarted","Data":"c7b0ea1a1d3a8176fa479dd879edc55a9854dd07375e0a213439e06488bfbd24"}
Jul 28 15:34:34.123635 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.123395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:34.128012 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.127988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb"
Jul 28 15:34:34.141491 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.141365    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v5pwb" podStartSLOduration=2.92488099 podStartE2EDuration="4.14134797s" totalImagesPullingTime="1.21646698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:34.140908831 +0000 UTC m=+102.141148755" watchObservedRunningTime="2026-07-28 15:34:34.14134797 +0000 UTC m=+102.141587895"
Jul 28 15:34:34.289772 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.289739    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:34:34.292913 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.292898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.295724 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.295701    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 15:34:34.295890 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.295868    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6rdhg\""
Jul 28 15:34:34.295953 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.295870    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 15:34:34.295994 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.295952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 15:34:34.296431 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.296418    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 15:34:34.296486 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.296467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 15:34:34.297196 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.297173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 15:34:34.297296 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.297223    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 15:34:34.301862 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.301843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:34:34.385062 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.384970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827dj\" (UniqueName: \"kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.385062 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.385007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.385062 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.385028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.385298 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.385080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.385298 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.385155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.385298 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.385193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487065 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487247 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487247 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827dj\" (UniqueName: \"kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487247 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487407 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.487407 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.487315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.490967 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.490937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.491126 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.491085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.491233 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.491216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.491291 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.491245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.492352 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.492325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.499422 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.499400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-827dj\" (UniqueName: \"kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj\") pod \"console-fc95f8b8d-85lrd\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") " pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.603072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.603038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:34.727866 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.727814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:34:34.728247 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:34.728220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.730578 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:34.730552    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ec0553_c095_4624_82ec_722f79895f2c.slice/crio-1795b224ce18a0d5b970cc42bc1a820254b0d889c2bb7106d3cbf3d14793748e WatchSource:0}: Error finding container 1795b224ce18a0d5b970cc42bc1a820254b0d889c2bb7106d3cbf3d14793748e: Status 404 returned error can't find the container with id 1795b224ce18a0d5b970cc42bc1a820254b0d889c2bb7106d3cbf3d14793748e
Jul 28 15:34:35.081800 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:35.081758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.128368 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:35.128334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc95f8b8d-85lrd" event={"ID":"01ec0553-c095-4624-82ec-722f79895f2c","Type":"ContainerStarted","Data":"1795b224ce18a0d5b970cc42bc1a820254b0d889c2bb7106d3cbf3d14793748e"}
Jul 28 15:34:35.182863 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:35.182825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.664205 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:36.664168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.764833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:36.764800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.830553 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:36.830521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:37.085775 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:37.085739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:37.184280 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:37.184247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:38.142574 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:38.142540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc95f8b8d-85lrd" event={"ID":"01ec0553-c095-4624-82ec-722f79895f2c","Type":"ContainerStarted","Data":"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"}
Jul 28 15:34:38.164508 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:38.164459    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fc95f8b8d-85lrd" podStartSLOduration=0.88128499 podStartE2EDuration="4.164440518s" totalImagesPullingTime="3.283155528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:38.163471972 +0000 UTC m=+106.163711897" watchObservedRunningTime="2026-07-28 15:34:38.164440518 +0000 UTC m=+106.164680442"
Jul 28 15:34:39.279420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:39.279388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:39.359526 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:39.359492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:39.699633 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:39.699541    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" podUID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" containerName="registry" containerID="cri-o://cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5" gracePeriod=30
Jul 28 15:34:39.948548 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:39.948523    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:40.027874 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.027874 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.027986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-466v5\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.028055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates\") pod \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\" (UID: \"9d0bf632-720d-4ada-bda7-1a2c61ac7f11\") "
Jul 28 15:34:40.028571 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.028516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:34:40.028665 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.028579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:34:40.030394 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.030364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:34:40.030394 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.030381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:34:40.030666 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.030648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:34:40.030710 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.030682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5" (OuterVolumeSpecName: "kube-api-access-466v5") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "kube-api-access-466v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:34:40.030851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.030834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:34:40.037293 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.037270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9d0bf632-720d-4ada-bda7-1a2c61ac7f11" (UID: "9d0bf632-720d-4ada-bda7-1a2c61ac7f11"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:34:40.128534 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128502    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-bound-sa-token\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128534 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128530    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-ca-trust-extracted\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128534 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128540    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-image-registry-private-configuration\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128550    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128559    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-installation-pull-secrets\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128567    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-trusted-ca\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128575    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466v5\" (UniqueName: \"kubernetes.io/projected/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-kube-api-access-466v5\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.128754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.128591    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d0bf632-720d-4ada-bda7-1a2c61ac7f11-registry-certificates\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:34:40.138365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.138324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:40.151179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.151149    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" containerID="cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5" exitCode=0
Jul 28 15:34:40.151286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.151197    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b778d4587-kcswv"
Jul 28 15:34:40.151286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.151225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" event={"ID":"9d0bf632-720d-4ada-bda7-1a2c61ac7f11","Type":"ContainerDied","Data":"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"}
Jul 28 15:34:40.151286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.151269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b778d4587-kcswv" event={"ID":"9d0bf632-720d-4ada-bda7-1a2c61ac7f11","Type":"ContainerDied","Data":"35c303dae8478935930c6f92e7bde4fa8051e3d42e9acb4b731635f9bb3d8b1c"}
Jul 28 15:34:40.151423 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.151292    2577 scope.go:117] "RemoveContainer" containerID="cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"
Jul 28 15:34:40.159268 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.159251    2577 scope.go:117] "RemoveContainer" containerID="cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"
Jul 28 15:34:40.159536 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:34:40.159515    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5\": container with ID starting with cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5 not found: ID does not exist" containerID="cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"
Jul 28 15:34:40.159594 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.159545    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5"} err="failed to get container status \"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5\": rpc error: code = NotFound desc = could not find container \"cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5\": container with ID starting with cfe841591e1a9daeceb2cd277a1fe9ce1efae6ee8292bbff84c6a799186c74c5 not found: ID does not exist"
Jul 28 15:34:40.172280 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.172254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:34:40.176109 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.176076    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b778d4587-kcswv"]
Jul 28 15:34:40.238554 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.238528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:40.561212 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:40.561179    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" path="/var/lib/kubelet/pods/9d0bf632-720d-4ada-bda7-1a2c61ac7f11/volumes"
Jul 28 15:34:42.666650 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.666618    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:34:42.667085 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.666840    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" containerName="registry"
Jul 28 15:34:42.667085 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.666850    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" containerName="registry"
Jul 28 15:34:42.667085 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.666892    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0bf632-720d-4ada-bda7-1a2c61ac7f11" containerName="registry"
Jul 28 15:34:42.669689 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.669674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.683553 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.683527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 15:34:42.684472 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.684441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:34:42.746888 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.746854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.746888 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.746891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.747110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.746910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szr9z\" (UniqueName: \"kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.747110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.746932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.747110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.747011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.747110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.747037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.747242 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.747148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847606 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847606 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szr9z\" (UniqueName: \"kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.847776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.847741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.848518 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.848493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.848645 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.848533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.848645 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.848621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.848645 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.848634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.850249 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.850220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.850249 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.850245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.856453 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.856432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szr9z\" (UniqueName: \"kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z\") pod \"console-76fb58b949-w64mw\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") " pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:42.978110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:42.978063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:43.107856 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:43.107817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:34:43.110844 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:34:43.110817    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a80cd3_d812_4c0c_8e23_422e6e60aae0.slice/crio-7799ba6009c478ddc80f483332bb22321042d38abec7af4501153c9b6f89268d WatchSource:0}: Error finding container 7799ba6009c478ddc80f483332bb22321042d38abec7af4501153c9b6f89268d: Status 404 returned error can't find the container with id 7799ba6009c478ddc80f483332bb22321042d38abec7af4501153c9b6f89268d
Jul 28 15:34:43.162792 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:43.162759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fb58b949-w64mw" event={"ID":"59a80cd3-d812-4c0c-8e23-422e6e60aae0","Type":"ContainerStarted","Data":"7799ba6009c478ddc80f483332bb22321042d38abec7af4501153c9b6f89268d"}
Jul 28 15:34:43.286416 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:43.286335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:43.386742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:43.386707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.167367 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:44.167330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fb58b949-w64mw" event={"ID":"59a80cd3-d812-4c0c-8e23-422e6e60aae0","Type":"ContainerStarted","Data":"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"}
Jul 28 15:34:44.187583 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:44.187537    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76fb58b949-w64mw" podStartSLOduration=2.18752108 podStartE2EDuration="2.18752108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:44.186159845 +0000 UTC m=+112.186399770" watchObservedRunningTime="2026-07-28 15:34:44.18752108 +0000 UTC m=+112.187761042"
Jul 28 15:34:44.604030 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:44.603995    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:44.604225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:44.604122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:44.610064 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:44.610037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:45.174899 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:45.174872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:34:46.182723 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:46.182690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:46.242860 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:46.242819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:47.942974 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:47.942938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:48.001642 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:48.001611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:50.774442 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:50.774409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:50.875149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:50.875076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:52.978930 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:52.978887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:52.979355 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:52.978949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:52.983747 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:52.983725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:53.201783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:53.201753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:34:53.257481 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:53.257392    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:34:53.311287 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:53.311244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:54.295636 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:54.295604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:54.370246 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:54.370202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.287019 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:55.286978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.350810 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:55.350767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.061041 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:56.061008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.161667 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:56.161631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.915655 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:56.915622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.994359 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:56.994326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:57.825632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:57.825591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:57.923683 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:57.923650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:58.023261 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:34:58.023229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:02.298721 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:02.298684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:35:02.300904 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:02.300883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e7bcf7-53f3-4c4b-97dd-5764f946f446-metrics-certs\") pod \"network-metrics-daemon-tk25s\" (UID: \"b9e7bcf7-53f3-4c4b-97dd-5764f946f446\") " pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:35:02.585875 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:02.585804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25sjx\""
Jul 28 15:35:02.593021 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:02.593003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk25s"
Jul 28 15:35:02.952796 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:02.952773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk25s"]
Jul 28 15:35:02.955330 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:35:02.955300    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e7bcf7_53f3_4c4b_97dd_5764f946f446.slice/crio-14d877ee428907dec59b41f93065de5ef8cf10e0c0340bb71196b9b73e29b14d WatchSource:0}: Error finding container 14d877ee428907dec59b41f93065de5ef8cf10e0c0340bb71196b9b73e29b14d: Status 404 returned error can't find the container with id 14d877ee428907dec59b41f93065de5ef8cf10e0c0340bb71196b9b73e29b14d
Jul 28 15:35:03.230484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:03.230446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk25s" event={"ID":"b9e7bcf7-53f3-4c4b-97dd-5764f946f446","Type":"ContainerStarted","Data":"14d877ee428907dec59b41f93065de5ef8cf10e0c0340bb71196b9b73e29b14d"}
Jul 28 15:35:04.235942 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:04.235902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk25s" event={"ID":"b9e7bcf7-53f3-4c4b-97dd-5764f946f446","Type":"ContainerStarted","Data":"e49ca41512f586920fecc71bdf474e2dd652ac4342a9ed42e68fb2e91dda0405"}
Jul 28 15:35:04.236413 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:04.235988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk25s" event={"ID":"b9e7bcf7-53f3-4c4b-97dd-5764f946f446","Type":"ContainerStarted","Data":"b106288e90880726403704fe7ef300f1e33ee8392761c54feded78737d7182fb"}
Jul 28 15:35:04.253776 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:04.253727    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tk25s" podStartSLOduration=130.345584991 podStartE2EDuration="2m11.253712042s" totalImagesPullingTime="908.127051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:04.252971842 +0000 UTC m=+132.253211768" watchObservedRunningTime="2026-07-28 15:35:04.253712042 +0000 UTC m=+132.253951969"
Jul 28 15:35:04.346393 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:04.346358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:04.412017 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:04.411981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:05.776450 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:05.776411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:05.881105 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:05.881073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:18.277127 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.277044    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fc95f8b8d-85lrd" podUID="01ec0553-c095-4624-82ec-722f79895f2c" containerName="console" containerID="cri-o://d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9" gracePeriod=15
Jul 28 15:35:18.523694 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.523670    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:35:18.622788 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.622788 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.622788 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-827dj\" (UniqueName: \"kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.623048 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.623048 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.623048 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.622863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca\") pod \"01ec0553-c095-4624-82ec-722f79895f2c\" (UID: \"01ec0553-c095-4624-82ec-722f79895f2c\") "
Jul 28 15:35:18.623327 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.623286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config" (OuterVolumeSpecName: "console-config") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:18.623327 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.623296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:18.623495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.623310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca" (OuterVolumeSpecName: "service-ca") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:18.625177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.625149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:18.625257 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.625168    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj" (OuterVolumeSpecName: "kube-api-access-827dj") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "kube-api-access-827dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:18.625309 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.625265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01ec0553-c095-4624-82ec-722f79895f2c" (UID: "01ec0553-c095-4624-82ec-722f79895f2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:18.723994 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.723964    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-console-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:18.723994 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.723991    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-oauth-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:18.723994 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.724000    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-service-ca\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:18.724224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.724008    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec0553-c095-4624-82ec-722f79895f2c-console-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:18.724224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.724017    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec0553-c095-4624-82ec-722f79895f2c-oauth-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:18.724224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:18.724025    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827dj\" (UniqueName: \"kubernetes.io/projected/01ec0553-c095-4624-82ec-722f79895f2c-kube-api-access-827dj\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:35:19.289380 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.289347    2577 generic.go:356] "Generic (PLEG): container finished" podID="01ec0553-c095-4624-82ec-722f79895f2c" containerID="d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9" exitCode=0
Jul 28 15:35:19.289790 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.289384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc95f8b8d-85lrd"
Jul 28 15:35:19.289790 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.289406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc95f8b8d-85lrd" event={"ID":"01ec0553-c095-4624-82ec-722f79895f2c","Type":"ContainerDied","Data":"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"}
Jul 28 15:35:19.289790 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.289449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc95f8b8d-85lrd" event={"ID":"01ec0553-c095-4624-82ec-722f79895f2c","Type":"ContainerDied","Data":"1795b224ce18a0d5b970cc42bc1a820254b0d889c2bb7106d3cbf3d14793748e"}
Jul 28 15:35:19.289790 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.289469    2577 scope.go:117] "RemoveContainer" containerID="d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"
Jul 28 15:35:19.297814 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.297796    2577 scope.go:117] "RemoveContainer" containerID="d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"
Jul 28 15:35:19.298086 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:35:19.298064    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9\": container with ID starting with d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9 not found: ID does not exist" containerID="d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"
Jul 28 15:35:19.298152 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.298111    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9"} err="failed to get container status \"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9\": rpc error: code = NotFound desc = could not find container \"d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9\": container with ID starting with d59ed73de8135958507df43d66bdf71daf41a6900b361e512dbd2d26e80635d9 not found: ID does not exist"
Jul 28 15:35:19.314089 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.314065    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:35:19.318350 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:19.318332    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fc95f8b8d-85lrd"]
Jul 28 15:35:20.561200 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:20.561171    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ec0553-c095-4624-82ec-722f79895f2c" path="/var/lib/kubelet/pods/01ec0553-c095-4624-82ec-722f79895f2c/volumes"
Jul 28 15:35:41.074774 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:41.074727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:45.316643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:45.316549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:46.157176 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:46.157144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:54.122824 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.122785    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:35:54.123282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.123151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec0553-c095-4624-82ec-722f79895f2c" containerName="console"
Jul 28 15:35:54.123282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.123168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec0553-c095-4624-82ec-722f79895f2c" containerName="console"
Jul 28 15:35:54.123282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.123238    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec0553-c095-4624-82ec-722f79895f2c" containerName="console"
Jul 28 15:35:54.128901 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.128877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.139706 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.139682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:35:54.208381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.208343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:54.282927 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.282887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283081 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.282934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283081 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.282957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5wk\" (UniqueName: \"kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283081 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.283008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283081 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.283067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283260 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.283156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.283260 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.283185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384564 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384564 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384564 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384763 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384763 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384883 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5wk\" (UniqueName: \"kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.384933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.384914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.385326 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.385301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.385447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.385395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.385447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.385439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.385625 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.385602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.387006 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.386986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.387197 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.387179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.394981 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.394956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5wk\" (UniqueName: \"kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk\") pod \"console-6f8db68f87-fh2p7\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") " pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.418493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.418464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:54.437429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.437402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:35:54.566708 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:54.566683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:35:54.568841 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:35:54.568814    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40d619d_5f0a_47cf_8c35_f92e39c334f0.slice/crio-6cf8a4724d11d31ce6a47359e8dff84724a0d5d5947163c7f87277fc91123ecf WatchSource:0}: Error finding container 6cf8a4724d11d31ce6a47359e8dff84724a0d5d5947163c7f87277fc91123ecf: Status 404 returned error can't find the container with id 6cf8a4724d11d31ce6a47359e8dff84724a0d5d5947163c7f87277fc91123ecf
Jul 28 15:35:55.292635 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:55.292600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:55.408552 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:55.408514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8db68f87-fh2p7" event={"ID":"e40d619d-5f0a-47cf-8c35-f92e39c334f0","Type":"ContainerStarted","Data":"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"}
Jul 28 15:35:55.408708 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:55.408568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8db68f87-fh2p7" event={"ID":"e40d619d-5f0a-47cf-8c35-f92e39c334f0","Type":"ContainerStarted","Data":"6cf8a4724d11d31ce6a47359e8dff84724a0d5d5947163c7f87277fc91123ecf"}
Jul 28 15:35:55.428393 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:35:55.428337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f8db68f87-fh2p7" podStartSLOduration=1.428322048 podStartE2EDuration="1.428322048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:55.427891523 +0000 UTC m=+183.428131462" watchObservedRunningTime="2026-07-28 15:35:55.428322048 +0000 UTC m=+183.428561972"
Jul 28 15:36:04.437641 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:04.437607    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:36:04.438036 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:04.437665    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:36:04.442573 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:04.442548    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:36:05.447042 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:05.447012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:36:05.497305 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:05.497271    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:36:05.615722 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:05.615693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:05.717272 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:05.717188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:05.821200 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:05.821167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:28.547263 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:28.547223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:30.519482 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.519441    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76fb58b949-w64mw" podUID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" containerName="console" containerID="cri-o://162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279" gracePeriod=15
Jul 28 15:36:30.768120 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.768079    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:36:30.830486 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830486 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830499    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szr9z\" (UniqueName: \"kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830687 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert\") pod \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\" (UID: \"59a80cd3-d812-4c0c-8e23-422e6e60aae0\") "
Jul 28 15:36:30.830983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:36:30.830983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830953    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca" (OuterVolumeSpecName: "service-ca") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:36:30.830983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config" (OuterVolumeSpecName: "console-config") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:36:30.830983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.830967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:36:30.832619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.832593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:36:30.832708 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.832654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z" (OuterVolumeSpecName: "kube-api-access-szr9z") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "kube-api-access-szr9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:36:30.832749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.832732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59a80cd3-d812-4c0c-8e23-422e6e60aae0" (UID: "59a80cd3-d812-4c0c-8e23-422e6e60aae0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:36:30.931425 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931386    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-service-ca\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931425 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931418    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-oauth-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931425 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931429    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szr9z\" (UniqueName: \"kubernetes.io/projected/59a80cd3-d812-4c0c-8e23-422e6e60aae0-kube-api-access-szr9z\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931439    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931448    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a80cd3-d812-4c0c-8e23-422e6e60aae0-trusted-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931456    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:30.931643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:30.931464    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a80cd3-d812-4c0c-8e23-422e6e60aae0-console-oauth-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:36:31.526789 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.526755    2577 generic.go:356] "Generic (PLEG): container finished" podID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" containerID="162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279" exitCode=0
Jul 28 15:36:31.527209 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.526802    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fb58b949-w64mw"
Jul 28 15:36:31.527209 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.526836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fb58b949-w64mw" event={"ID":"59a80cd3-d812-4c0c-8e23-422e6e60aae0","Type":"ContainerDied","Data":"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"}
Jul 28 15:36:31.527209 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.526904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fb58b949-w64mw" event={"ID":"59a80cd3-d812-4c0c-8e23-422e6e60aae0","Type":"ContainerDied","Data":"7799ba6009c478ddc80f483332bb22321042d38abec7af4501153c9b6f89268d"}
Jul 28 15:36:31.527209 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.526925    2577 scope.go:117] "RemoveContainer" containerID="162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"
Jul 28 15:36:31.535347 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.535299    2577 scope.go:117] "RemoveContainer" containerID="162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"
Jul 28 15:36:31.535601 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:36:31.535578    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279\": container with ID starting with 162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279 not found: ID does not exist" containerID="162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"
Jul 28 15:36:31.535694 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.535608    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279"} err="failed to get container status \"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279\": rpc error: code = NotFound desc = could not find container \"162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279\": container with ID starting with 162ad6ec89b6776660032b0e56884c486a30f35c6c220176a729caca350e2279 not found: ID does not exist"
Jul 28 15:36:31.555300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.555272    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:36:31.557303 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:31.557281    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76fb58b949-w64mw"]
Jul 28 15:36:32.560835 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:32.560792    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" path="/var/lib/kubelet/pods/59a80cd3-d812-4c0c-8e23-422e6e60aae0/volumes"
Jul 28 15:36:45.386640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:45.386604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:45.451910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:36:45.451884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:00.574999 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:00.574957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:00.641673 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:00.641641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:12.610569 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:12.610531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:12.674750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:12.674717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:20.780283 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:20.780210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:20.880600 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:20.880565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:20.942658 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:20.942624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:25.776383 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.776349    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:37:25.776755 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.776592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" containerName="console"
Jul 28 15:37:25.776755 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.776604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" containerName="console"
Jul 28 15:37:25.776755 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.776643    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a80cd3-d812-4c0c-8e23-422e6e60aae0" containerName="console"
Jul 28 15:37:25.779214 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.779196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.790696 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.790566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:37:25.847305 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.847274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:25.910671 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.910636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:25.915718 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckpbd\" (UniqueName: \"kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915878 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915918 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915951 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.915985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.915959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:25.988169 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:25.988134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:26.017017 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.016971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017017 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckpbd\" (UniqueName: \"kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017278 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017477 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.017895 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.018083 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.018083 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.017937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.018224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.018063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.019692 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.019672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.019775 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.019709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.025387 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.025364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckpbd\" (UniqueName: \"kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd\") pod \"console-68f4dfdbb7-j2gln\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") " pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.049985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.049925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:26.088115 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.088063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:26.214239 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.214207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:37:26.217438 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:37:26.217416    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb92af83_50d9_4fba_a8fd_2e8c3ee5c3a7.slice/crio-3f22b57d4c9e905d0f0f1401d7c143937eee9570e4c5b50770c3350cbd2cb539 WatchSource:0}: Error finding container 3f22b57d4c9e905d0f0f1401d7c143937eee9570e4c5b50770c3350cbd2cb539: Status 404 returned error can't find the container with id 3f22b57d4c9e905d0f0f1401d7c143937eee9570e4c5b50770c3350cbd2cb539
Jul 28 15:37:26.708471 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.708428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f4dfdbb7-j2gln" event={"ID":"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7","Type":"ContainerStarted","Data":"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"}
Jul 28 15:37:26.708641 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.708487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f4dfdbb7-j2gln" event={"ID":"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7","Type":"ContainerStarted","Data":"3f22b57d4c9e905d0f0f1401d7c143937eee9570e4c5b50770c3350cbd2cb539"}
Jul 28 15:37:26.727935 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.727885    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f4dfdbb7-j2gln" podStartSLOduration=1.727870496 podStartE2EDuration="1.727870496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:37:26.726722818 +0000 UTC m=+274.726962784" watchObservedRunningTime="2026-07-28 15:37:26.727870496 +0000 UTC m=+274.728110421"
Jul 28 15:37:26.792117 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.792062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:26.858504 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:26.858468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:36.088230 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.088185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:36.088763 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.088265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:36.093197 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.093179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:36.744429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.744399    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:37:36.798679 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.798647    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:37:36.944208 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:36.944174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.046346 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:37.046270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.113543 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:37.113514    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:52.360108 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:52.360063    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:37:52.363029 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:52.363006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:37:52.366482 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:37:52.366461    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 15:38:01.818787 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:01.818729    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f8db68f87-fh2p7" podUID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" containerName="console" containerID="cri-o://91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065" gracePeriod=15
Jul 28 15:38:02.059605 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.059579    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:38:02.084796 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084796 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd5wk\" (UniqueName: \"kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.084993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.084942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert\") pod \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\" (UID: \"e40d619d-5f0a-47cf-8c35-f92e39c334f0\") "
Jul 28 15:38:02.085268 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.085181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:38:02.085662 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.085602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca" (OuterVolumeSpecName: "service-ca") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:38:02.086184 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.086157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:38:02.086333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.086313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config" (OuterVolumeSpecName: "console-config") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:38:02.087242 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.087216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:38:02.087448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.087427    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk" (OuterVolumeSpecName: "kube-api-access-wd5wk") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "kube-api-access-wd5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:38:02.087528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.087513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e40d619d-5f0a-47cf-8c35-f92e39c334f0" (UID: "e40d619d-5f0a-47cf-8c35-f92e39c334f0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:38:02.186170 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186132    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-trusted-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186170 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186167    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-service-ca\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186180    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186191    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-oauth-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186202    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-oauth-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186214    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5wk\" (UniqueName: \"kubernetes.io/projected/e40d619d-5f0a-47cf-8c35-f92e39c334f0-kube-api-access-wd5wk\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.186357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.186225    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e40d619d-5f0a-47cf-8c35-f92e39c334f0-console-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:38:02.828037 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.827949    2577 generic.go:356] "Generic (PLEG): container finished" podID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" containerID="91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065" exitCode=0
Jul 28 15:38:02.828037 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.828006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8db68f87-fh2p7" event={"ID":"e40d619d-5f0a-47cf-8c35-f92e39c334f0","Type":"ContainerDied","Data":"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"}
Jul 28 15:38:02.828037 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.828012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8db68f87-fh2p7"
Jul 28 15:38:02.828584 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.828047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8db68f87-fh2p7" event={"ID":"e40d619d-5f0a-47cf-8c35-f92e39c334f0","Type":"ContainerDied","Data":"6cf8a4724d11d31ce6a47359e8dff84724a0d5d5947163c7f87277fc91123ecf"}
Jul 28 15:38:02.828584 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.828075    2577 scope.go:117] "RemoveContainer" containerID="91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"
Jul 28 15:38:02.835961 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.835943    2577 scope.go:117] "RemoveContainer" containerID="91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"
Jul 28 15:38:02.836223 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:38:02.836200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065\": container with ID starting with 91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065 not found: ID does not exist" containerID="91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"
Jul 28 15:38:02.836321 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.836228    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065"} err="failed to get container status \"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065\": rpc error: code = NotFound desc = could not find container \"91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065\": container with ID starting with 91dd87055eda052f6cd2f4e82f8c81796d6f3193b5de1e03eaea0c818daeb065 not found: ID does not exist"
Jul 28 15:38:02.849446 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.849425    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:38:02.856073 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:02.856054    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f8db68f87-fh2p7"]
Jul 28 15:38:04.560629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:04.560596    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" path="/var/lib/kubelet/pods/e40d619d-5f0a-47cf-8c35-f92e39c334f0/volumes"
Jul 28 15:38:19.831315 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:19.831276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:21.361555 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:21.361516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:42.633783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:42.633748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:42.757215 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:42.757179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:47.926319 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:47.926245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.227661 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.227631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.485062 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.484988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.594383 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.594322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.699603 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.699566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.801825 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.801739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.899932 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:48.899902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:49.020024 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:49.019991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:49.183674 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:49.183592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:49.355618 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:49.355581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:49.451989 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:38:49.451917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:02.565392 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:02.565358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:05.187236 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.187201    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:05.187593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.187436    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" containerName="console"
Jul 28 15:40:05.187593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.187446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" containerName="console"
Jul 28 15:40:05.187593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.187493    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e40d619d-5f0a-47cf-8c35-f92e39c334f0" containerName="console"
Jul 28 15:40:05.190188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.190170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.192892 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.192869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 15:40:05.193149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.193131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 15:40:05.193218 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.193180    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 15:40:05.194579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.194561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7jfzx\""
Jul 28 15:40:05.196159 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.196138    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tvg6g"]
Jul 28 15:40:05.199258 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.199239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.202175 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.202110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kswqn\""
Jul 28 15:40:05.202175 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.202130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 15:40:05.202402 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.202385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:05.212448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.212413    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tvg6g"]
Jul 28 15:40:05.216988 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.216963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:05.240954 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.240912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.241112 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.240965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxrp\" (UniqueName: \"kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.241112 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.240988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqpf\" (UniqueName: \"kubernetes.io/projected/c53c25d0-9847-41eb-9318-da1e4addbbd6-kube-api-access-8zqpf\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.241112 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.241026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.309622 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.309582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:05.341923 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.341890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.342045 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.341944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxrp\" (UniqueName: \"kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.342045 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.341966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqpf\" (UniqueName: \"kubernetes.io/projected/c53c25d0-9847-41eb-9318-da1e4addbbd6-kube-api-access-8zqpf\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.342045 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.341985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.342189 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:40:05.342082    2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 15:40:05.342189 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:40:05.342170    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert podName:c53c25d0-9847-41eb-9318-da1e4addbbd6 nodeName:}" failed. No retries permitted until 2026-07-28 15:40:05.84215511 +0000 UTC m=+433.842395012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert") pod "llmisvc-controller-manager-84fc959947-tvg6g" (UID: "c53c25d0-9847-41eb-9318-da1e4addbbd6") : secret "llmisvc-webhook-server-cert" not found
Jul 28 15:40:05.344388 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.344364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.352617 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.352590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxrp\" (UniqueName: \"kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp\") pod \"kserve-controller-manager-85d8b8ddfc-xhvgz\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.352733 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.352640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqpf\" (UniqueName: \"kubernetes.io/projected/c53c25d0-9847-41eb-9318-da1e4addbbd6-kube-api-access-8zqpf\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.500715 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.500679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:05.611813 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.611786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:05.630545 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.630440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:05.633004 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:40:05.632978    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a9d137_26ce_49b6_9026_f911cbf8fcf6.slice/crio-bd24f23aad0e2ce7b3dd194a9031e47251afeefd094299a7d6fcbc6d1e7bdc8e WatchSource:0}: Error finding container bd24f23aad0e2ce7b3dd194a9031e47251afeefd094299a7d6fcbc6d1e7bdc8e: Status 404 returned error can't find the container with id bd24f23aad0e2ce7b3dd194a9031e47251afeefd094299a7d6fcbc6d1e7bdc8e
Jul 28 15:40:05.634188 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.634171    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:40:05.846181 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.846077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:05.848453 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:05.848414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53c25d0-9847-41eb-9318-da1e4addbbd6-cert\") pod \"llmisvc-controller-manager-84fc959947-tvg6g\" (UID: \"c53c25d0-9847-41eb-9318-da1e4addbbd6\") " pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:06.004381 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:06.004337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:06.102820 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:06.102736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:06.115106 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:06.114966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:06.237545 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:06.237503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" event={"ID":"90a9d137-26ce-49b6-9026-f911cbf8fcf6","Type":"ContainerStarted","Data":"bd24f23aad0e2ce7b3dd194a9031e47251afeefd094299a7d6fcbc6d1e7bdc8e"}
Jul 28 15:40:06.273537 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:06.273506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tvg6g"]
Jul 28 15:40:06.275487 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:40:06.275455    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc53c25d0_9847_41eb_9318_da1e4addbbd6.slice/crio-cb533a8ea204bd5add043e31f436acfda745ae85dc1074798e7849647f0547eb WatchSource:0}: Error finding container cb533a8ea204bd5add043e31f436acfda745ae85dc1074798e7849647f0547eb: Status 404 returned error can't find the container with id cb533a8ea204bd5add043e31f436acfda745ae85dc1074798e7849647f0547eb
Jul 28 15:40:07.243818 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:07.243783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g" event={"ID":"c53c25d0-9847-41eb-9318-da1e4addbbd6","Type":"ContainerStarted","Data":"cb533a8ea204bd5add043e31f436acfda745ae85dc1074798e7849647f0547eb"}
Jul 28 15:40:10.256912 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.256875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g" event={"ID":"c53c25d0-9847-41eb-9318-da1e4addbbd6","Type":"ContainerStarted","Data":"a7eb8c65b85031a4bb7992121c731105b78d6c1e1fa21e5512e5626580483b9f"}
Jul 28 15:40:10.257369 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.256939    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:10.258345 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.258325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" event={"ID":"90a9d137-26ce-49b6-9026-f911cbf8fcf6","Type":"ContainerStarted","Data":"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"}
Jul 28 15:40:10.258470 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.258453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:10.273365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.273325    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g" podStartSLOduration=1.983996972 podStartE2EDuration="5.273311365s" totalImagesPullingTime="3.289314393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:10.272661808 +0000 UTC m=+438.272901758" watchObservedRunningTime="2026-07-28 15:40:10.273311365 +0000 UTC m=+438.273551290"
Jul 28 15:40:10.290741 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:10.290697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" podStartSLOduration=2.214150525 podStartE2EDuration="5.290683386s" totalImagesPullingTime="3.076532861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:10.28979913 +0000 UTC m=+438.290039074" watchObservedRunningTime="2026-07-28 15:40:10.290683386 +0000 UTC m=+438.290923313"
Jul 28 15:40:23.996809 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:23.996729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:41.263401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:41.263363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-tvg6g"
Jul 28 15:40:41.266417 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:41.266395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:48.902348 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:48.902307    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:48.902824 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:48.902555    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" podUID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" containerName="manager" containerID="cri-o://ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338" gracePeriod=10
Jul 28 15:40:48.935970 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:48.935938    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"]
Jul 28 15:40:48.937916 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:48.937900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:48.952179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:48.952151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"]
Jul 28 15:40:49.080757 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.080715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-cert\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.080891 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.080778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hv8v\" (UniqueName: \"kubernetes.io/projected/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-kube-api-access-4hv8v\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.139881 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.139858    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:49.182252 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.182170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv8v\" (UniqueName: \"kubernetes.io/projected/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-kube-api-access-4hv8v\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.182393 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.182272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-cert\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.184557 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.184526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-cert\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.191762 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.191738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hv8v\" (UniqueName: \"kubernetes.io/projected/53d1f1f3-1f1d-4963-89f8-77b2b89ea812-kube-api-access-4hv8v\") pod \"kserve-controller-manager-85d8b8ddfc-fcnvf\" (UID: \"53d1f1f3-1f1d-4963-89f8-77b2b89ea812\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.282938 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.282884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert\") pod \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") "
Jul 28 15:40:49.283161 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.282964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnxrp\" (UniqueName: \"kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp\") pod \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\" (UID: \"90a9d137-26ce-49b6-9026-f911cbf8fcf6\") "
Jul 28 15:40:49.285179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.285146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert" (OuterVolumeSpecName: "cert") pod "90a9d137-26ce-49b6-9026-f911cbf8fcf6" (UID: "90a9d137-26ce-49b6-9026-f911cbf8fcf6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:40:49.285179 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.285151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp" (OuterVolumeSpecName: "kube-api-access-nnxrp") pod "90a9d137-26ce-49b6-9026-f911cbf8fcf6" (UID: "90a9d137-26ce-49b6-9026-f911cbf8fcf6"). InnerVolumeSpecName "kube-api-access-nnxrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:40:49.287262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.287244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:49.383855 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.383819    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90a9d137-26ce-49b6-9026-f911cbf8fcf6-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:40:49.383983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.383859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxrp\" (UniqueName: \"kubernetes.io/projected/90a9d137-26ce-49b6-9026-f911cbf8fcf6-kube-api-access-nnxrp\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:40:49.390131 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.390081    2577 generic.go:356] "Generic (PLEG): container finished" podID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" containerID="ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338" exitCode=0
Jul 28 15:40:49.390300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.390149    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"
Jul 28 15:40:49.390300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.390168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" event={"ID":"90a9d137-26ce-49b6-9026-f911cbf8fcf6","Type":"ContainerDied","Data":"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"}
Jul 28 15:40:49.390300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.390232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-xhvgz" event={"ID":"90a9d137-26ce-49b6-9026-f911cbf8fcf6","Type":"ContainerDied","Data":"bd24f23aad0e2ce7b3dd194a9031e47251afeefd094299a7d6fcbc6d1e7bdc8e"}
Jul 28 15:40:49.390300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.390262    2577 scope.go:117] "RemoveContainer" containerID="ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"
Jul 28 15:40:49.398556 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.398467    2577 scope.go:117] "RemoveContainer" containerID="ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"
Jul 28 15:40:49.398870 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:40:49.398846    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338\": container with ID starting with ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338 not found: ID does not exist" containerID="ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"
Jul 28 15:40:49.398966 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.398876    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338"} err="failed to get container status \"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338\": rpc error: code = NotFound desc = could not find container \"ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338\": container with ID starting with ba06092038c906ce94c06aba050aeff46a93c32065bf7d1e0b98f18538f1a338 not found: ID does not exist"
Jul 28 15:40:49.411702 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.411674    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:49.414418 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.414396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"]
Jul 28 15:40:49.417376 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:49.417353    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-xhvgz"]
Jul 28 15:40:49.417623 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:40:49.417607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d1f1f3_1f1d_4963_89f8_77b2b89ea812.slice/crio-ed0b0c4df84316e8c87f1190cd6ff2151db8d4955b3f14b95b4c76cfc30ba9a5 WatchSource:0}: Error finding container ed0b0c4df84316e8c87f1190cd6ff2151db8d4955b3f14b95b4c76cfc30ba9a5: Status 404 returned error can't find the container with id ed0b0c4df84316e8c87f1190cd6ff2151db8d4955b3f14b95b4c76cfc30ba9a5
Jul 28 15:40:50.394478 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:50.394442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf" event={"ID":"53d1f1f3-1f1d-4963-89f8-77b2b89ea812","Type":"ContainerStarted","Data":"a5ab0552c5eca138da74db73e213cd40f06d9904e1a79d5765bf4247bb7c9f6a"}
Jul 28 15:40:50.394929 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:50.394496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf" event={"ID":"53d1f1f3-1f1d-4963-89f8-77b2b89ea812","Type":"ContainerStarted","Data":"ed0b0c4df84316e8c87f1190cd6ff2151db8d4955b3f14b95b4c76cfc30ba9a5"}
Jul 28 15:40:50.394929 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:50.394544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:40:50.412668 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:50.412618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf" podStartSLOduration=2.094174786 podStartE2EDuration="2.412600489s" totalImagesPullingTime="318.425703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:50.411930731 +0000 UTC m=+478.412170690" watchObservedRunningTime="2026-07-28 15:40:50.412600489 +0000 UTC m=+478.412840414"
Jul 28 15:40:50.563263 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:40:50.561820    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" path="/var/lib/kubelet/pods/90a9d137-26ce-49b6-9026-f911cbf8fcf6/volumes"
Jul 28 15:41:08.489608 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:08.489544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:08.913476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:08.913390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:21.403275 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:21.403236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-fcnvf"
Jul 28 15:41:22.360044 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.360007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:22.395830 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.395800    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s6m9w"]
Jul 28 15:41:22.396059 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.396040    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" containerName="manager"
Jul 28 15:41:22.396059 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.396053    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" containerName="manager"
Jul 28 15:41:22.396174 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.396123    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a9d137-26ce-49b6-9026-f911cbf8fcf6" containerName="manager"
Jul 28 15:41:22.398971 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.398956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.401763 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.401733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 15:41:22.401763 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.401738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bdqkw\""
Jul 28 15:41:22.410925 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.410904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s6m9w"]
Jul 28 15:41:22.455890 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.455859    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:22.519248 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.519213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.519544 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.519523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mmk\" (UniqueName: \"kubernetes.io/projected/98133422-887a-4247-ad15-139acc514e74-kube-api-access-m2mmk\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.620235 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.620153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mmk\" (UniqueName: \"kubernetes.io/projected/98133422-887a-4247-ad15-139acc514e74-kube-api-access-m2mmk\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.620235 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.620219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.620418 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:41:22.620312    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 15:41:22.620418 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:41:22.620384    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert podName:98133422-887a-4247-ad15-139acc514e74 nodeName:}" failed. No retries permitted until 2026-07-28 15:41:23.120364666 +0000 UTC m=+511.120604569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert") pod "odh-model-controller-75d9cdbddf-s6m9w" (UID: "98133422-887a-4247-ad15-139acc514e74") : secret "odh-model-controller-webhook-cert" not found
Jul 28 15:41:22.631312 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.631286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mmk\" (UniqueName: \"kubernetes.io/projected/98133422-887a-4247-ad15-139acc514e74-kube-api-access-m2mmk\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:22.660697 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.660665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:22.791628 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:22.791591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:23.124605 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.124566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:23.127050 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.127020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98133422-887a-4247-ad15-139acc514e74-cert\") pod \"odh-model-controller-75d9cdbddf-s6m9w\" (UID: \"98133422-887a-4247-ad15-139acc514e74\") " pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:23.147051 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.147029    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:23.219513 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.219479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:23.308404 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.308373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:23.438800 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.438766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s6m9w"]
Jul 28 15:41:23.442222 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:41:23.442198    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98133422_887a_4247_ad15_139acc514e74.slice/crio-f18f4b3ac625cbb45b86f816f794276dbab0124554025a24b3dbe7e3d2f13c2f WatchSource:0}: Error finding container f18f4b3ac625cbb45b86f816f794276dbab0124554025a24b3dbe7e3d2f13c2f: Status 404 returned error can't find the container with id f18f4b3ac625cbb45b86f816f794276dbab0124554025a24b3dbe7e3d2f13c2f
Jul 28 15:41:23.505640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:23.505604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w" event={"ID":"98133422-887a-4247-ad15-139acc514e74","Type":"ContainerStarted","Data":"f18f4b3ac625cbb45b86f816f794276dbab0124554025a24b3dbe7e3d2f13c2f"}
Jul 28 15:41:26.518469 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.518435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w" event={"ID":"98133422-887a-4247-ad15-139acc514e74","Type":"ContainerStarted","Data":"7aab6ef7ab243f35c2de153d7415a5e73e529bef5054fae709cb060a4d5b420a"}
Jul 28 15:41:26.518841 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.518550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:26.535739 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.535714    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cdd4cd54-vs4gm"]
Jul 28 15:41:26.538424 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.538383    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w" podStartSLOduration=2.017093216 podStartE2EDuration="4.538370138s" totalImagesPullingTime="2.521276922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:26.537712505 +0000 UTC m=+514.537952432" watchObservedRunningTime="2026-07-28 15:41:26.538370138 +0000 UTC m=+514.538610063"
Jul 28 15:41:26.538963 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.538915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.550909 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.550887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdd4cd54-vs4gm"]
Jul 28 15:41:26.551087 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-console-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551174 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-service-ca\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551213 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-oauth-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551213 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551305 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zbr\" (UniqueName: \"kubernetes.io/projected/80690136-e0fa-4675-a09a-b6b727afe242-kube-api-access-h5zbr\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551357 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-oauth-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.551429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.551415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-trusted-ca-bundle\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.599849 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.599819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:26.651742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-console-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.651742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-service-ca\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.651922 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-oauth-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.651922 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.651922 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zbr\" (UniqueName: \"kubernetes.io/projected/80690136-e0fa-4675-a09a-b6b727afe242-kube-api-access-h5zbr\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.651922 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.651906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-oauth-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.652138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.652002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-trusted-ca-bundle\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.652547 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.652524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-console-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.652679 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.652627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-service-ca\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.652769 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.652740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-oauth-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.652993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.652974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80690136-e0fa-4675-a09a-b6b727afe242-trusted-ca-bundle\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.654573 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.654550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-oauth-config\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.654655 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.654575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80690136-e0fa-4675-a09a-b6b727afe242-console-serving-cert\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.660722 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.660696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zbr\" (UniqueName: \"kubernetes.io/projected/80690136-e0fa-4675-a09a-b6b727afe242-kube-api-access-h5zbr\") pod \"console-5cdd4cd54-vs4gm\" (UID: \"80690136-e0fa-4675-a09a-b6b727afe242\") " pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.680066 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.680042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:26.848891 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.848820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:26.974378 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:26.974348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdd4cd54-vs4gm"]
Jul 28 15:41:27.523039 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:27.522998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdd4cd54-vs4gm" event={"ID":"80690136-e0fa-4675-a09a-b6b727afe242","Type":"ContainerStarted","Data":"ca15dda5634616beae36ca5f71e0cc37eb594a8f24d7f3c50485b96113cea5d6"}
Jul 28 15:41:27.523420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:27.523058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdd4cd54-vs4gm" event={"ID":"80690136-e0fa-4675-a09a-b6b727afe242","Type":"ContainerStarted","Data":"489cb902a226146322594180164c9e6e53c0cf5ee985fdbbf7f2d3ef6fe90883"}
Jul 28 15:41:27.543032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:27.542981    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cdd4cd54-vs4gm" podStartSLOduration=1.542966134 podStartE2EDuration="1.542966134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:27.542291597 +0000 UTC m=+515.542531555" watchObservedRunningTime="2026-07-28 15:41:27.542966134 +0000 UTC m=+515.543206058"
Jul 28 15:41:36.849352 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:36.849310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:36.849769 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:36.849379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:36.854016 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:36.853990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:37.525281 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.525251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-s6m9w"
Jul 28 15:41:37.563522 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.563494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cdd4cd54-vs4gm"
Jul 28 15:41:37.617316 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.617278    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:41:37.735897 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.735851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:37.839309 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.839234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:37.940660 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:41:37.940627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:02.639595 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.639554    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68f4dfdbb7-j2gln" podUID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" containerName="console" containerID="cri-o://a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f" gracePeriod=15
Jul 28 15:42:02.878993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.878964    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:42:02.918742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918742 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.918993 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.918914    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckpbd\" (UniqueName: \"kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd\") pod \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\" (UID: \"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7\") "
Jul 28 15:42:02.919234 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.919207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config" (OuterVolumeSpecName: "console-config") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:42:02.919324 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.919302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:42:02.919391 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.919371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:42:02.919475 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.919448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:42:02.920939 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.920915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:42:02.921049 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.920918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd" (OuterVolumeSpecName: "kube-api-access-ckpbd") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "kube-api-access-ckpbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:42:02.921049 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:02.920935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" (UID: "fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:42:03.020070 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020040    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020070 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020064    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-trusted-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020070 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020075    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020084    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-console-oauth-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020112    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-service-ca\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020121    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-oauth-serving-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.020282 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.020129    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckpbd\" (UniqueName: \"kubernetes.io/projected/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7-kube-api-access-ckpbd\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:42:03.651057 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.651022    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" containerID="a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f" exitCode=0
Jul 28 15:42:03.651473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.651064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f4dfdbb7-j2gln" event={"ID":"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7","Type":"ContainerDied","Data":"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"}
Jul 28 15:42:03.651473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.651069    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f4dfdbb7-j2gln"
Jul 28 15:42:03.651473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.651120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f4dfdbb7-j2gln" event={"ID":"fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7","Type":"ContainerDied","Data":"3f22b57d4c9e905d0f0f1401d7c143937eee9570e4c5b50770c3350cbd2cb539"}
Jul 28 15:42:03.651473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.651142    2577 scope.go:117] "RemoveContainer" containerID="a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"
Jul 28 15:42:03.663759 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.663616    2577 scope.go:117] "RemoveContainer" containerID="a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"
Jul 28 15:42:03.664071 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:42:03.664047    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f\": container with ID starting with a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f not found: ID does not exist" containerID="a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"
Jul 28 15:42:03.664193 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.664078    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f"} err="failed to get container status \"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f\": rpc error: code = NotFound desc = could not find container \"a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f\": container with ID starting with a64487392def7163d03ed99dd7f0d104fe3eee82e549e096ee4592c4a65a246f not found: ID does not exist"
Jul 28 15:42:03.681057 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.681031    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:42:03.684873 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:03.684850    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f4dfdbb7-j2gln"]
Jul 28 15:42:04.561274 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:04.561239    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" path="/var/lib/kubelet/pods/fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7/volumes"
Jul 28 15:42:11.104176 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:11.104140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:52.385530 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:52.385500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:42:52.386392 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:52.386373    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:42:57.482071 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:57.482032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:57.581753 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:42:57.581698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.136878 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.136829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.446001 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.445902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.568619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.568577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.569084 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.569067    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:43:09.569333 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.569321    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" containerName="console"
Jul 28 15:43:09.569376 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.569335    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" containerName="console"
Jul 28 15:43:09.569408 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.569383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb92af83-50d9-4fba-a8fd-2e8c3ee5c3a7" containerName="console"
Jul 28 15:43:09.572209 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.572189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.575122 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.575084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8fc0c-predictor-serving-cert\""
Jul 28 15:43:09.575338 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.575161    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:43:09.575493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.575220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s454d\""
Jul 28 15:43:09.576401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.576385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:43:09.576505 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.576406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8fc0c-kube-rbac-proxy-sar-config\""
Jul 28 15:43:09.581670 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.581648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:43:09.671782 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.671748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.693503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.693453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.693667 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.693589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vdr\" (UniqueName: \"kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.693667 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.693623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.767491 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.767452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.793933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.793898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.793933 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.793935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vdr\" (UniqueName: \"kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.794212 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.793961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.794212 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:43:09.794145    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-serving-cert: secret "error-404-isvc-8fc0c-predictor-serving-cert" not found
Jul 28 15:43:09.794363 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:43:09.794216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls podName:1b9a48a5-6386-453b-b01a-2f6de330d82f nodeName:}" failed. No retries permitted until 2026-07-28 15:43:10.294196334 +0000 UTC m=+618.294436238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls") pod "error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" (UID: "1b9a48a5-6386-453b-b01a-2f6de330d82f") : secret "error-404-isvc-8fc0c-predictor-serving-cert" not found
Jul 28 15:43:09.794565 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.794545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.804434 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.804402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vdr\" (UniqueName: \"kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:09.861883 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.861841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.894758 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.894726    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:43:09.898017 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.898001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:09.901473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.901452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 28 15:43:09.901604 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.901484    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 28 15:43:09.901604 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.901484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:43:09.901604 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.901455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:43:09.909047 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.909021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:43:09.939719 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.939692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:09.995331 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.995303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:09.995448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.995337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd654\" (UniqueName: \"kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:09.995448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.995355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:09.995448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.995416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:09.995448 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:09.995441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd654\" (UniqueName: \"kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096527 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096527 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.096746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.096714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.097132 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.097088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.097192 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.097088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.098739 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.098721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.105171 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.105150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd654\" (UniqueName: \"kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.109559 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.109443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:10.209797 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.209766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:10.297591 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.297563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:10.300445 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.300413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") pod \"error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") " pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:10.348301 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.348221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:43:10.351247 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:43:10.351220    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8f4c23_7816_4b84_a079_74ae156a87a6.slice/crio-d56300f79ffe186bfa8b99250e4e3db32d972f71deb600c76806106356ad4831 WatchSource:0}: Error finding container d56300f79ffe186bfa8b99250e4e3db32d972f71deb600c76806106356ad4831: Status 404 returned error can't find the container with id d56300f79ffe186bfa8b99250e4e3db32d972f71deb600c76806106356ad4831
Jul 28 15:43:10.483484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.483447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:10.613595 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.613513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:43:10.616267 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:43:10.616232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9a48a5_6386_453b_b01a_2f6de330d82f.slice/crio-06aadfdb42d7f923a39810116e4c785b3ef494ea1e0ac1e29197c7ef5e7b5c49 WatchSource:0}: Error finding container 06aadfdb42d7f923a39810116e4c785b3ef494ea1e0ac1e29197c7ef5e7b5c49: Status 404 returned error can't find the container with id 06aadfdb42d7f923a39810116e4c785b3ef494ea1e0ac1e29197c7ef5e7b5c49
Jul 28 15:43:10.883196 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.883110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerStarted","Data":"06aadfdb42d7f923a39810116e4c785b3ef494ea1e0ac1e29197c7ef5e7b5c49"}
Jul 28 15:43:10.884262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:10.884235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerStarted","Data":"d56300f79ffe186bfa8b99250e4e3db32d972f71deb600c76806106356ad4831"}
Jul 28 15:43:23.947217 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:23.947127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerStarted","Data":"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"}
Jul 28 15:43:23.948826 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:23.948795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerStarted","Data":"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"}
Jul 28 15:43:26.963566 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:26.963515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerStarted","Data":"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"}
Jul 28 15:43:26.963960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:26.963738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:26.963960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:26.963868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:26.965250 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:26.965215    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:43:26.981939 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:26.981897    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podStartSLOduration=2.70744699 podStartE2EDuration="17.981860969s" totalImagesPullingTime="15.274413979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:26.980796796 +0000 UTC m=+634.981036763" watchObservedRunningTime="2026-07-28 15:43:26.981860969 +0000 UTC m=+634.982100973"
Jul 28 15:43:27.968699 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:27.968664    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerID="29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933" exitCode=0
Jul 28 15:43:27.969187 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:27.968739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerDied","Data":"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"}
Jul 28 15:43:27.969187 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:27.969073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:43:29.646289 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:29.646168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:32.974479 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:32.974444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:43:32.974946 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:32.974922    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:43:33.996518 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:33.996474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerStarted","Data":"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"}
Jul 28 15:43:33.996910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:33.996558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerStarted","Data":"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"}
Jul 28 15:43:33.996910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:33.996861    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:34.019489 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:34.019444    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podStartSLOduration=2.394381917 podStartE2EDuration="25.019431194s" totalImagesPullingTime="18.725670019s" totalInitContainerRuntime="3.899379258s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:34.018535059 +0000 UTC m=+642.018774984" watchObservedRunningTime="2026-07-28 15:43:34.019431194 +0000 UTC m=+642.019671119"
Jul 28 15:43:35.000056 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:35.000023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:35.001216 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:35.001191    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:43:36.003974 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:36.003940    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:43:41.008790 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:41.008758    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:43:41.009389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:41.009358    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:43:42.975285 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:42.975243    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:43:51.009374 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:51.009330    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:43:52.975900 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:43:52.975854    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:44:01.010295 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:01.010254    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:44:02.974883 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:02.974833    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 15:44:11.009888 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:11.009842    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:44:12.976162 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:12.976130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:44:21.009803 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:21.009763    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:44:31.009698 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:31.009659    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:44:39.173068 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.173030    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:44:39.175478 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.175450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.178390 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.178367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8fc0c-kube-rbac-proxy-sar-config\""
Jul 28 15:44:39.178495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.178404    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8fc0c-serving-cert\""
Jul 28 15:44:39.183753 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.183729    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:44:39.190758 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.190724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:39.289281 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.289253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.289456 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.289301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.367245 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.367194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:39.389853 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.389829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.389980 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.389876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.390535 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.390515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.392212 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.392183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls\") pod \"switch-graph-8fc0c-b5cb6cc48-tdw5s\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") " pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.486236 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.486216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:39.609037 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:39.609016    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:44:39.611314 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:44:39.611292    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c77117_f53a_48db_9b2a_fd649c404509.slice/crio-a0b221a5170e4d9b309f7cba1cad1d5c605433cffd05516c2e0f560ddb5ee9a5 WatchSource:0}: Error finding container a0b221a5170e4d9b309f7cba1cad1d5c605433cffd05516c2e0f560ddb5ee9a5: Status 404 returned error can't find the container with id a0b221a5170e4d9b309f7cba1cad1d5c605433cffd05516c2e0f560ddb5ee9a5
Jul 28 15:44:40.230874 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:40.230836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" event={"ID":"72c77117-f53a-48db-9b2a-fd649c404509","Type":"ContainerStarted","Data":"a0b221a5170e4d9b309f7cba1cad1d5c605433cffd05516c2e0f560ddb5ee9a5"}
Jul 28 15:44:41.010050 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:41.010003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:44:42.239906 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:42.239866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" event={"ID":"72c77117-f53a-48db-9b2a-fd649c404509","Type":"ContainerStarted","Data":"94a1b9d4a4b35bdf7d7cd0c7866d5b622122dc65aea8e7d59e243489a0fff2ea"}
Jul 28 15:44:42.240429 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:42.240084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:42.257018 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:42.256967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podStartSLOduration=1.111153683 podStartE2EDuration="3.256949959s" totalImagesPullingTime="2.145796276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:44:42.256572522 +0000 UTC m=+710.256812448" watchObservedRunningTime="2026-07-28 15:44:42.256949959 +0000 UTC m=+710.257189884"
Jul 28 15:44:48.249816 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:48.249740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:44:49.290004 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.289967    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:44:49.290447 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.290213    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" containerID="cri-o://94a1b9d4a4b35bdf7d7cd0c7866d5b622122dc65aea8e7d59e243489a0fff2ea" gracePeriod=30
Jul 28 15:44:49.332707 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.332675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:49.501170 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.501139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:49.603320 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.603231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:49.623516 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.623482    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:44:49.625648 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.625631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.629615 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.629590    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b277-predictor-serving-cert\""
Jul 28 15:44:49.629748 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.629594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b277-kube-rbac-proxy-sar-config\""
Jul 28 15:44:49.641436 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.641412    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:44:49.667666 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.667639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.667820 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.667681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvxr\" (UniqueName: \"kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.667887 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.667814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.692846 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.692816    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:44:49.693043 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.693020    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container" containerID="cri-o://0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1" gracePeriod=30
Jul 28 15:44:49.693144 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.693062    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kube-rbac-proxy" containerID="cri-o://14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0" gracePeriod=30
Jul 28 15:44:49.711482 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.711457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:49.769121 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.769071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.769318 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.769135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvxr\" (UniqueName: \"kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.769318 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.769186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.769848 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.769816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.771598 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.771578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.781138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.781116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvxr\" (UniqueName: \"kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr\") pod \"error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") " pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:49.813420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.813393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:49.936073 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:49.935989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:50.063693 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.063660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:44:50.066260 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:44:50.066227    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda191d9c3_8d75_496b_867f_026862812283.slice/crio-860eebc06234c0bb431c8e950868f452185a4a68eafdd8ae2c6e8ff4328973b0 WatchSource:0}: Error finding container 860eebc06234c0bb431c8e950868f452185a4a68eafdd8ae2c6e8ff4328973b0: Status 404 returned error can't find the container with id 860eebc06234c0bb431c8e950868f452185a4a68eafdd8ae2c6e8ff4328973b0
Jul 28 15:44:50.271433 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.271399    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerID="14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0" exitCode=2
Jul 28 15:44:50.271602 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.271478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerDied","Data":"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"}
Jul 28 15:44:50.273224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.273198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerStarted","Data":"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"}
Jul 28 15:44:50.273378 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.273244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerStarted","Data":"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"}
Jul 28 15:44:50.273378 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.273265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerStarted","Data":"860eebc06234c0bb431c8e950868f452185a4a68eafdd8ae2c6e8ff4328973b0"}
Jul 28 15:44:50.273494 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.273410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:50.294752 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:50.294707    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podStartSLOduration=1.294694674 podStartE2EDuration="1.294694674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:44:50.293348464 +0000 UTC m=+718.293588412" watchObservedRunningTime="2026-07-28 15:44:50.294694674 +0000 UTC m=+718.294934674"
Jul 28 15:44:51.009982 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:51.009953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:44:51.277214 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:51.277120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:51.278424 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:51.278398    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:44:52.281267 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.281220    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:44:52.837248 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.837221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:44:52.894397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.894320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vdr\" (UniqueName: \"kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr\") pod \"1b9a48a5-6386-453b-b01a-2f6de330d82f\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") "
Jul 28 15:44:52.894397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.894361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config\") pod \"1b9a48a5-6386-453b-b01a-2f6de330d82f\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") "
Jul 28 15:44:52.894397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.894395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") pod \"1b9a48a5-6386-453b-b01a-2f6de330d82f\" (UID: \"1b9a48a5-6386-453b-b01a-2f6de330d82f\") "
Jul 28 15:44:52.894666 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.894641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b9a48a5-6386-453b-b01a-2f6de330d82f" (UID: "1b9a48a5-6386-453b-b01a-2f6de330d82f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:44:52.896489 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.896468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b9a48a5-6386-453b-b01a-2f6de330d82f" (UID: "1b9a48a5-6386-453b-b01a-2f6de330d82f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:44:52.896612 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.896587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr" (OuterVolumeSpecName: "kube-api-access-m8vdr") pod "1b9a48a5-6386-453b-b01a-2f6de330d82f" (UID: "1b9a48a5-6386-453b-b01a-2f6de330d82f"). InnerVolumeSpecName "kube-api-access-m8vdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:44:52.995047 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.995011    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vdr\" (UniqueName: \"kubernetes.io/projected/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-api-access-m8vdr\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:44:52.995047 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.995040    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b9a48a5-6386-453b-b01a-2f6de330d82f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:44:52.995047 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:52.995050    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b9a48a5-6386-453b-b01a-2f6de330d82f-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:44:53.249139 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.249075    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:53.286511 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.286475    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerID="0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1" exitCode=0
Jul 28 15:44:53.286954 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.286541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"
Jul 28 15:44:53.286954 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.286563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerDied","Data":"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"}
Jul 28 15:44:53.286954 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.286647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp" event={"ID":"1b9a48a5-6386-453b-b01a-2f6de330d82f","Type":"ContainerDied","Data":"06aadfdb42d7f923a39810116e4c785b3ef494ea1e0ac1e29197c7ef5e7b5c49"}
Jul 28 15:44:53.286954 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.286687    2577 scope.go:117] "RemoveContainer" containerID="14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"
Jul 28 15:44:53.295363 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.295344    2577 scope.go:117] "RemoveContainer" containerID="0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"
Jul 28 15:44:53.303279 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.303258    2577 scope.go:117] "RemoveContainer" containerID="14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"
Jul 28 15:44:53.303537 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:44:53.303519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0\": container with ID starting with 14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0 not found: ID does not exist" containerID="14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"
Jul 28 15:44:53.303619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.303550    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0"} err="failed to get container status \"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0\": rpc error: code = NotFound desc = could not find container \"14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0\": container with ID starting with 14387d15bbc02fda180f18d6e3de653281d948a75e4960874b362608efaf6ec0 not found: ID does not exist"
Jul 28 15:44:53.303619 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.303573    2577 scope.go:117] "RemoveContainer" containerID="0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"
Jul 28 15:44:53.303800 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:44:53.303783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1\": container with ID starting with 0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1 not found: ID does not exist" containerID="0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"
Jul 28 15:44:53.303835 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.303807    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1"} err="failed to get container status \"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1\": rpc error: code = NotFound desc = could not find container \"0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1\": container with ID starting with 0f105c88c97b1d7d58d410fb68ebaa6f1a826f7b344910861eafbee1d8976af1 not found: ID does not exist"
Jul 28 15:44:53.309748 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.309718    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:44:53.314042 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:53.314016    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8fc0c-predictor-548ddb8fdb-56zdp"]
Jul 28 15:44:54.560885 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:54.560849    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" path="/var/lib/kubelet/pods/1b9a48a5-6386-453b-b01a-2f6de330d82f/volumes"
Jul 28 15:44:57.285726 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:57.285694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:44:57.286198 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:57.286164    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:44:58.248900 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:44:58.248857    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:03.248344 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:03.248305    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:03.248813 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:03.248410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:45:06.287141 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:06.287082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:07.286925 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:07.286883    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:45:08.249004 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:08.248958    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:13.248850 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:13.248802    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:17.286825 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:17.286787    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:45:18.248857 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:18.248809    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:19.230113 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230069    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:19.230501 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230412    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container"
Jul 28 15:45:19.230501 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230424    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container"
Jul 28 15:45:19.230501 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kube-rbac-proxy"
Jul 28 15:45:19.230501 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kube-rbac-proxy"
Jul 28 15:45:19.230668 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230525    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kserve-container"
Jul 28 15:45:19.230668 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.230536    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9a48a5-6386-453b-b01a-2f6de330d82f" containerName="kube-rbac-proxy"
Jul 28 15:45:19.233285 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.233267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.236297 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.236263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 15:45:19.236297 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.236292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 15:45:19.241656 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.241628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:19.254061 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.254035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:19.391038 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.391005    2577 generic.go:356] "Generic (PLEG): container finished" podID="72c77117-f53a-48db-9b2a-fd649c404509" containerID="94a1b9d4a4b35bdf7d7cd0c7866d5b622122dc65aea8e7d59e243489a0fff2ea" exitCode=0
Jul 28 15:45:19.391214 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.391082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" event={"ID":"72c77117-f53a-48db-9b2a-fd649c404509","Type":"ContainerDied","Data":"94a1b9d4a4b35bdf7d7cd0c7866d5b622122dc65aea8e7d59e243489a0fff2ea"}
Jul 28 15:45:19.406207 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.406172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.406350 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.406242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.441802 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.441775    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:45:19.507476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.507390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.507476 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.507447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.507702 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:19.507570    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 28 15:45:19.507702 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:19.507628    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls podName:bf6990a5-7a7f-4dc8-8b2f-f88d42575a47 nodeName:}" failed. No retries permitted until 2026-07-28 15:45:20.007611264 +0000 UTC m=+748.007851190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls") pod "model-chainer-5578dcbdf9-55mvt" (UID: "bf6990a5-7a7f-4dc8-8b2f-f88d42575a47") : secret "model-chainer-serving-cert" not found
Jul 28 15:45:19.508143 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.508120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:19.608572 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.608538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls\") pod \"72c77117-f53a-48db-9b2a-fd649c404509\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") "
Jul 28 15:45:19.608767 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.608618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle\") pod \"72c77117-f53a-48db-9b2a-fd649c404509\" (UID: \"72c77117-f53a-48db-9b2a-fd649c404509\") "
Jul 28 15:45:19.608979 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.608953    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "72c77117-f53a-48db-9b2a-fd649c404509" (UID: "72c77117-f53a-48db-9b2a-fd649c404509"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:19.610651 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.610627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72c77117-f53a-48db-9b2a-fd649c404509" (UID: "72c77117-f53a-48db-9b2a-fd649c404509"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:19.672716 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.672681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:19.710316 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.710284    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c77117-f53a-48db-9b2a-fd649c404509-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:19.710316 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:19.710315    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72c77117-f53a-48db-9b2a-fd649c404509-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:20.013004 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.012968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:20.015450 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.015427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") pod \"model-chainer-5578dcbdf9-55mvt\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") " pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:20.144199 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.144163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:20.271261 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.271171    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:20.274732 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:45:20.274693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6990a5_7a7f_4dc8_8b2f_f88d42575a47.slice/crio-71bcab3c1c98996348a3bb423485e6ecf6e92e5176ceeb5a78798b741681055b WatchSource:0}: Error finding container 71bcab3c1c98996348a3bb423485e6ecf6e92e5176ceeb5a78798b741681055b: Status 404 returned error can't find the container with id 71bcab3c1c98996348a3bb423485e6ecf6e92e5176ceeb5a78798b741681055b
Jul 28 15:45:20.396386 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.396351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" event={"ID":"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47","Type":"ContainerStarted","Data":"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"}
Jul 28 15:45:20.396569 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.396408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" event={"ID":"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47","Type":"ContainerStarted","Data":"71bcab3c1c98996348a3bb423485e6ecf6e92e5176ceeb5a78798b741681055b"}
Jul 28 15:45:20.396569 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.396542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:20.397545 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.397523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s" event={"ID":"72c77117-f53a-48db-9b2a-fd649c404509","Type":"ContainerDied","Data":"a0b221a5170e4d9b309f7cba1cad1d5c605433cffd05516c2e0f560ddb5ee9a5"}
Jul 28 15:45:20.397545 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.397541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"
Jul 28 15:45:20.397708 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.397566    2577 scope.go:117] "RemoveContainer" containerID="94a1b9d4a4b35bdf7d7cd0c7866d5b622122dc65aea8e7d59e243489a0fff2ea"
Jul 28 15:45:20.413845 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.413787    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podStartSLOduration=1.4137734229999999 podStartE2EDuration="1.413773423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:20.412691657 +0000 UTC m=+748.412931582" watchObservedRunningTime="2026-07-28 15:45:20.413773423 +0000 UTC m=+748.414013347"
Jul 28 15:45:20.427904 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.427882    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:45:20.431865 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.431845    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fc0c-b5cb6cc48-tdw5s"]
Jul 28 15:45:20.561465 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:20.561382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c77117-f53a-48db-9b2a-fd649c404509" path="/var/lib/kubelet/pods/72c77117-f53a-48db-9b2a-fd649c404509/volumes"
Jul 28 15:45:26.409463 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:26.409431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:27.287114 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:27.287054    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:45:29.317225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.317193    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:29.317654 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.317400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" containerID="cri-o://db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1" gracePeriod=30
Jul 28 15:45:29.363729 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.363696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:29.442782 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.442742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:45:29.443076 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.443027    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" containerID="cri-o://80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd" gracePeriod=30
Jul 28 15:45:29.443240 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.443031    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kube-rbac-proxy" containerID="cri-o://68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1" gracePeriod=30
Jul 28 15:45:29.489025 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.488986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:29.583418 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.583340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:29.645327 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.645290    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:45:29.645754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.645734    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c"
Jul 28 15:45:29.645860 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.645757    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c"
Jul 28 15:45:29.645860 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.645823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c77117-f53a-48db-9b2a-fd649c404509" containerName="switch-graph-8fc0c"
Jul 28 15:45:29.649147 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.649122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.651957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.651933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-642bb-predictor-serving-cert\""
Jul 28 15:45:29.652087 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.651984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-642bb-kube-rbac-proxy-sar-config\""
Jul 28 15:45:29.657640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.657616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:45:29.684060 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.684030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.684221 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.684067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.684221 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.684088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrl7\" (UniqueName: \"kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.703818 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.703785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:29.784473 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.784437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrl7\" (UniqueName: \"kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.784649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.784538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.784649 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.784566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.784758 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:29.784668    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-642bb-predictor-serving-cert: secret "error-404-isvc-642bb-predictor-serving-cert" not found
Jul 28 15:45:29.784758 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:29.784732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls podName:babee5aa-d467-4143-892e-bbd46556aa8e nodeName:}" failed. No retries permitted until 2026-07-28 15:45:30.284711111 +0000 UTC m=+758.284951013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls") pod "error-404-isvc-642bb-predictor-6c6846787d-fmpcw" (UID: "babee5aa-d467-4143-892e-bbd46556aa8e") : secret "error-404-isvc-642bb-predictor-serving-cert" not found
Jul 28 15:45:29.785295 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.785270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:29.795493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:29.795471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrl7\" (UniqueName: \"kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:30.143957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.143909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:30.245006 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.244972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:30.289259 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.289220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:30.291642 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.291618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") pod \"error-404-isvc-642bb-predictor-6c6846787d-fmpcw\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") " pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:30.439768 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.439687    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerID="68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1" exitCode=2
Jul 28 15:45:30.440120 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.439762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerDied","Data":"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"}
Jul 28 15:45:30.559582 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.559556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:30.692326 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:30.692244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:45:30.695189 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:45:30.695159    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabee5aa_d467_4143_892e_bbd46556aa8e.slice/crio-c03e551e347b361604287bdbdd778bd61d220e7e02005154ec39e1f8f8cfb75c WatchSource:0}: Error finding container c03e551e347b361604287bdbdd778bd61d220e7e02005154ec39e1f8f8cfb75c: Status 404 returned error can't find the container with id c03e551e347b361604287bdbdd778bd61d220e7e02005154ec39e1f8f8cfb75c
Jul 28 15:45:31.004956 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.004912    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 28 15:45:31.009502 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.009472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 15:45:31.406507 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.406406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:31.447587 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.447540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerStarted","Data":"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"}
Jul 28 15:45:31.447960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.447630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:31.447960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.447665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerStarted","Data":"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"}
Jul 28 15:45:31.447960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.447696    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:31.447960 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.447725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerStarted","Data":"c03e551e347b361604287bdbdd778bd61d220e7e02005154ec39e1f8f8cfb75c"}
Jul 28 15:45:31.448926 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.448899    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:45:31.477460 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:31.477396    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podStartSLOduration=2.477378685 podStartE2EDuration="2.477378685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:31.476445107 +0000 UTC m=+759.476685031" watchObservedRunningTime="2026-07-28 15:45:31.477378685 +0000 UTC m=+759.477618610"
Jul 28 15:45:32.451450 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:32.451413    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:45:35.091746 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.091721    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:45:35.124905 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.124878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd654\" (UniqueName: \"kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654\") pod \"cb8f4c23-7816-4b84-a079-74ae156a87a6\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") "
Jul 28 15:45:35.125058 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.124917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config\") pod \"cb8f4c23-7816-4b84-a079-74ae156a87a6\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") "
Jul 28 15:45:35.125058 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.124943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert\") pod \"cb8f4c23-7816-4b84-a079-74ae156a87a6\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") "
Jul 28 15:45:35.125058 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.124979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location\") pod \"cb8f4c23-7816-4b84-a079-74ae156a87a6\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") "
Jul 28 15:45:35.125058 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.125015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls\") pod \"cb8f4c23-7816-4b84-a079-74ae156a87a6\" (UID: \"cb8f4c23-7816-4b84-a079-74ae156a87a6\") "
Jul 28 15:45:35.125365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.125309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb8f4c23-7816-4b84-a079-74ae156a87a6" (UID: "cb8f4c23-7816-4b84-a079-74ae156a87a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:45:35.125365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.125324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb8f4c23-7816-4b84-a079-74ae156a87a6" (UID: "cb8f4c23-7816-4b84-a079-74ae156a87a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:35.125365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.125341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb8f4c23-7816-4b84-a079-74ae156a87a6" (UID: "cb8f4c23-7816-4b84-a079-74ae156a87a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:35.126963 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.126941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654" (OuterVolumeSpecName: "kube-api-access-jd654") pod "cb8f4c23-7816-4b84-a079-74ae156a87a6" (UID: "cb8f4c23-7816-4b84-a079-74ae156a87a6"). InnerVolumeSpecName "kube-api-access-jd654". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:45:35.127223 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.127197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb8f4c23-7816-4b84-a079-74ae156a87a6" (UID: "cb8f4c23-7816-4b84-a079-74ae156a87a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:35.225528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.225443    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb8f4c23-7816-4b84-a079-74ae156a87a6-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.225528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.225472    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd654\" (UniqueName: \"kubernetes.io/projected/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-api-access-jd654\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.225528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.225482    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.225528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.225491    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb8f4c23-7816-4b84-a079-74ae156a87a6-cabundle-cert\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.225528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.225501    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb8f4c23-7816-4b84-a079-74ae156a87a6-kserve-provision-location\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.464047 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.464006    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerID="80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd" exitCode=0
Jul 28 15:45:35.464225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.464054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerDied","Data":"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"}
Jul 28 15:45:35.464225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.464067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"
Jul 28 15:45:35.464225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.464122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k" event={"ID":"cb8f4c23-7816-4b84-a079-74ae156a87a6","Type":"ContainerDied","Data":"d56300f79ffe186bfa8b99250e4e3db32d972f71deb600c76806106356ad4831"}
Jul 28 15:45:35.464225 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.464149    2577 scope.go:117] "RemoveContainer" containerID="68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"
Jul 28 15:45:35.473205 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.473184    2577 scope.go:117] "RemoveContainer" containerID="80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"
Jul 28 15:45:35.480339 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.480318    2577 scope.go:117] "RemoveContainer" containerID="29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"
Jul 28 15:45:35.488737 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.488686    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:45:35.489277 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.489255    2577 scope.go:117] "RemoveContainer" containerID="68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:35.489556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1\": container with ID starting with 68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1 not found: ID does not exist" containerID="68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.489591    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1"} err="failed to get container status \"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1\": rpc error: code = NotFound desc = could not find container \"68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1\": container with ID starting with 68c61ad1e408ce49ae5d6a542f8f10065dc76ceaa3f30bdc8526fe03e4a988b1 not found: ID does not exist"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.489615    2577 scope.go:117] "RemoveContainer" containerID="80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:35.489834    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd\": container with ID starting with 80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd not found: ID does not exist" containerID="80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.489855    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd"} err="failed to get container status \"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd\": rpc error: code = NotFound desc = could not find container \"80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd\": container with ID starting with 80db887ce9fbf6ed41a074d928b150b3d833548e9f95100be740ed0e66ad41bd not found: ID does not exist"
Jul 28 15:45:35.490020 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.489872    2577 scope.go:117] "RemoveContainer" containerID="29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"
Jul 28 15:45:35.490557 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:35.490283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933\": container with ID starting with 29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933 not found: ID does not exist" containerID="29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"
Jul 28 15:45:35.490557 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.490309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933"} err="failed to get container status \"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933\": rpc error: code = NotFound desc = could not find container \"29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933\": container with ID starting with 29add010855856a346ee9dab6054b9f67b2c26ce2f65ea2e011bb17122526933 not found: ID does not exist"
Jul 28 15:45:35.492935 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:35.492881    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-j7z2k"]
Jul 28 15:45:36.406565 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:36.406528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:36.560944 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:36.560908    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" path="/var/lib/kubelet/pods/cb8f4c23-7816-4b84-a079-74ae156a87a6/volumes"
Jul 28 15:45:37.287270 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:37.287240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:45:37.456083 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:37.456056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:45:37.456659 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:37.456631    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:45:41.406527 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:41.406486    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:41.406910 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:41.406588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:46.406750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:46.406709    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:47.457074 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:47.457029    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:45:49.559957 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.559925    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560229    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560242    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560259    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kube-rbac-proxy"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560266    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kube-rbac-proxy"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560273    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="storage-initializer"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560279    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="storage-initializer"
Jul 28 15:45:49.560323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560322    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kserve-container"
Jul 28 15:45:49.560545 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.560332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8f4c23-7816-4b84-a079-74ae156a87a6" containerName="kube-rbac-proxy"
Jul 28 15:45:49.563110 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.563078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.565770 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.565744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4b277-kube-rbac-proxy-sar-config\""
Jul 28 15:45:49.565895 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.565768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4b277-serving-cert\""
Jul 28 15:45:49.567121 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.567083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:49.573444 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.573420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:45:49.619456 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.619417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.619644 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.619473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.720408 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.720377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.720583 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.720425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.720583 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:49.720539    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-4b277-serving-cert: secret "switch-graph-4b277-serving-cert" not found
Jul 28 15:45:49.720678 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:49.720592    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls podName:43e5bece-01b6-40cf-a368-c1278d0afa4e nodeName:}" failed. No retries permitted until 2026-07-28 15:45:50.220576931 +0000 UTC m=+778.220816834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls") pod "switch-graph-4b277-54cd4bc6c7-w7b2h" (UID: "43e5bece-01b6-40cf-a368-c1278d0afa4e") : secret "switch-graph-4b277-serving-cert" not found
Jul 28 15:45:49.720996 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.720975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:49.844499 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:49.844384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:50.223607 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:50.223512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:50.225878 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:50.225855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") pod \"switch-graph-4b277-54cd4bc6c7-w7b2h\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") " pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:50.474218 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:50.474124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:50.606412 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:50.606378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:45:50.609789 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:45:50.609765    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e5bece_01b6_40cf_a368_c1278d0afa4e.slice/crio-0e1ed74ef14ecc894ee526c1c376756f13b24f4b65bedb77382fe31b1f64a4d3 WatchSource:0}: Error finding container 0e1ed74ef14ecc894ee526c1c376756f13b24f4b65bedb77382fe31b1f64a4d3: Status 404 returned error can't find the container with id 0e1ed74ef14ecc894ee526c1c376756f13b24f4b65bedb77382fe31b1f64a4d3
Jul 28 15:45:51.407153 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:51.407111    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:51.524773 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:51.524734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" event={"ID":"43e5bece-01b6-40cf-a368-c1278d0afa4e","Type":"ContainerStarted","Data":"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"}
Jul 28 15:45:51.524936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:51.524789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" event={"ID":"43e5bece-01b6-40cf-a368-c1278d0afa4e","Type":"ContainerStarted","Data":"0e1ed74ef14ecc894ee526c1c376756f13b24f4b65bedb77382fe31b1f64a4d3"}
Jul 28 15:45:51.524936 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:51.524820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:51.542695 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:51.542644    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podStartSLOduration=2.542630074 podStartE2EDuration="2.542630074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:51.541756344 +0000 UTC m=+779.541996289" watchObservedRunningTime="2026-07-28 15:45:51.542630074 +0000 UTC m=+779.542869998"
Jul 28 15:45:56.406927 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:56.406887    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:57.457150 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:57.457086    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:45:57.535940 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:57.535903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:45:59.453436 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.453408    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:59.490223 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.490196    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") pod \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") "
Jul 28 15:45:59.490364 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.490253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle\") pod \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\" (UID: \"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47\") "
Jul 28 15:45:59.490590 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.490566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" (UID: "bf6990a5-7a7f-4dc8-8b2f-f88d42575a47"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:59.492226 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.492201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" (UID: "bf6990a5-7a7f-4dc8-8b2f-f88d42575a47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:59.560979 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.560939    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerID="db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1" exitCode=0
Jul 28 15:45:59.561138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.560998    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"
Jul 28 15:45:59.561138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.561026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" event={"ID":"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47","Type":"ContainerDied","Data":"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"}
Jul 28 15:45:59.561138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.561087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt" event={"ID":"bf6990a5-7a7f-4dc8-8b2f-f88d42575a47","Type":"ContainerDied","Data":"71bcab3c1c98996348a3bb423485e6ecf6e92e5176ceeb5a78798b741681055b"}
Jul 28 15:45:59.561138 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.561131    2577 scope.go:117] "RemoveContainer" containerID="db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"
Jul 28 15:45:59.569671 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.569651    2577 scope.go:117] "RemoveContainer" containerID="db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"
Jul 28 15:45:59.569905 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:45:59.569886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1\": container with ID starting with db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1 not found: ID does not exist" containerID="db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"
Jul 28 15:45:59.569959 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.569913    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1"} err="failed to get container status \"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1\": rpc error: code = NotFound desc = could not find container \"db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1\": container with ID starting with db232c626e338c3b7b147f316a7a99ab93db8833d5dc26f89ea410c35ef62fe1 not found: ID does not exist"
Jul 28 15:45:59.582511 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.582464    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:59.586624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.586606    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5578dcbdf9-55mvt"]
Jul 28 15:45:59.591611 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.591587    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:45:59.591611 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:45:59.591607    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:46:00.560671 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:00.560638    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" path="/var/lib/kubelet/pods/bf6990a5-7a7f-4dc8-8b2f-f88d42575a47/volumes"
Jul 28 15:46:07.457400 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:07.457357    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:46:17.458074 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:17.457992    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:46:27.011808 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:27.011769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:27.076835 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:27.076798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:29.548268 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.548232    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:46:29.548629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.548526    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer"
Jul 28 15:46:29.548629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.548538    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer"
Jul 28 15:46:29.548629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.548603    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6990a5-7a7f-4dc8-8b2f-f88d42575a47" containerName="model-chainer"
Jul 28 15:46:29.551498 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.551481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:29.554469 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.554448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-642bb-serving-cert\""
Jul 28 15:46:29.554565 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.554448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-642bb-kube-rbac-proxy-sar-config\""
Jul 28 15:46:29.560719 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.560694    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:46:29.624070 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.624032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:29.624254 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.624089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:29.724879 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.724844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:29.725052 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.724900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:29.725052 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:46:29.725004    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-642bb-serving-cert: secret "sequence-graph-642bb-serving-cert" not found
Jul 28 15:46:29.725176 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:46:29.725119    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls podName:180c1216-cc9b-4404-afc1-9d206f5bbeb2 nodeName:}" failed. No retries permitted until 2026-07-28 15:46:30.225070505 +0000 UTC m=+818.225310419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls") pod "sequence-graph-642bb-65f5dc7d74-5xcjh" (UID: "180c1216-cc9b-4404-afc1-9d206f5bbeb2") : secret "sequence-graph-642bb-serving-cert" not found
Jul 28 15:46:29.725578 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:29.725558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:30.058723 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.058690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:30.155365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.155334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:30.228643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.228610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:30.230979 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.230958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") pod \"sequence-graph-642bb-65f5dc7d74-5xcjh\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") " pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:30.461658 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.461566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:30.587947 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.587924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:46:30.590344 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:46:30.590318    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180c1216_cc9b_4404_afc1_9d206f5bbeb2.slice/crio-0d9663900cd66ce3f33c86e577955152a4d32a780d767c156c73fae0a129c0ee WatchSource:0}: Error finding container 0d9663900cd66ce3f33c86e577955152a4d32a780d767c156c73fae0a129c0ee: Status 404 returned error can't find the container with id 0d9663900cd66ce3f33c86e577955152a4d32a780d767c156c73fae0a129c0ee
Jul 28 15:46:30.672733 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.672700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" event={"ID":"180c1216-cc9b-4404-afc1-9d206f5bbeb2","Type":"ContainerStarted","Data":"4947c48a45c748f2a91dd398bfa8287561c9c4e05dee17ec3234320b3e0e2660"}
Jul 28 15:46:30.672887 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.672756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" event={"ID":"180c1216-cc9b-4404-afc1-9d206f5bbeb2","Type":"ContainerStarted","Data":"0d9663900cd66ce3f33c86e577955152a4d32a780d767c156c73fae0a129c0ee"}
Jul 28 15:46:30.672887 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.672850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:46:30.691400 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:30.691348    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podStartSLOduration=1.69133193 podStartE2EDuration="1.69133193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:46:30.689971197 +0000 UTC m=+818.690211133" watchObservedRunningTime="2026-07-28 15:46:30.69133193 +0000 UTC m=+818.691571855"
Jul 28 15:46:36.682105 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:46:36.682073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:47:52.408435 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:47:52.408353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:47:52.408962 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:47:52.408518    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:49:16.107178 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:49:16.107069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:50:57.977244 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:50:57.977207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:50:58.062825 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:50:58.062789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:06.470625 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:51:06.470589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:52:45.453782 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:52:45.453733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:52:52.429916 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:52:52.429881    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:52:52.430466 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:52:52.430368    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:54:14.248869 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.248837    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:54:14.251415 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.249119    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" containerID="cri-o://5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab" gracePeriod=30
Jul 28 15:54:14.290535 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.290500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.448112 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.448075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.542725 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.542639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.614516 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.614480    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:54:14.617754 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.617732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.620706 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.620685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad35d-predictor-serving-cert\""
Jul 28 15:54:14.620881 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.620689    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad35d-kube-rbac-proxy-sar-config\""
Jul 28 15:54:14.633304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.633277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:54:14.643816 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.643787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.691818 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.691789    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:54:14.692053 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.692027    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" containerID="cri-o://7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41" gracePeriod=30
Jul 28 15:54:14.692144 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.692066    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kube-rbac-proxy" containerID="cri-o://aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10" gracePeriod=30
Jul 28 15:54:14.717390 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.717358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.717515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.717426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.717579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.717515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x76j\" (UniqueName: \"kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.750637 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.750608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.818618 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.818592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.818749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.818636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x76j\" (UniqueName: \"kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.818749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.818742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.819214 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.819194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.821221 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.821187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.831858 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.831836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x76j\" (UniqueName: \"kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j\") pod \"error-404-isvc-ad35d-predictor-6cb9647b76-schlh\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") " pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.852266 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.852237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:14.927548 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.927520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:14.956985 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:14.956958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:15.072793 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.072722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:54:15.075773 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:54:15.075744    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5cb8ed_0736_43af_b3b4_07b38ab90dc2.slice/crio-342cc8759b7c495f96119a12304ba031dd955f5ba8758bac932e35bf22a24a45 WatchSource:0}: Error finding container 342cc8759b7c495f96119a12304ba031dd955f5ba8758bac932e35bf22a24a45: Status 404 returned error can't find the container with id 342cc8759b7c495f96119a12304ba031dd955f5ba8758bac932e35bf22a24a45
Jul 28 15:54:15.348273 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.348183    2577 generic.go:356] "Generic (PLEG): container finished" podID="a191d9c3-8d75-496b-867f-026862812283" containerID="aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10" exitCode=2
Jul 28 15:54:15.348686 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.348262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerDied","Data":"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"}
Jul 28 15:54:15.349904 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.349876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerStarted","Data":"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"}
Jul 28 15:54:15.350049 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.349934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerStarted","Data":"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"}
Jul 28 15:54:15.350049 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.349966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerStarted","Data":"342cc8759b7c495f96119a12304ba031dd955f5ba8758bac932e35bf22a24a45"}
Jul 28 15:54:15.350049 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.350000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:15.374914 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:15.374869    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podStartSLOduration=1.37485481 podStartE2EDuration="1.37485481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:54:15.37402699 +0000 UTC m=+1283.374266910" watchObservedRunningTime="2026-07-28 15:54:15.37485481 +0000 UTC m=+1283.375094735"
Jul 28 15:54:16.353914 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:16.353881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:16.355059 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:16.355032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:17.282401 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:17.282357    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 15:54:17.286681 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:17.286655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 15:54:17.357408 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:17.357375    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:17.533970 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:17.533883    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:18.049480 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.049451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:54:18.146052 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.145965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config\") pod \"a191d9c3-8d75-496b-867f-026862812283\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") "
Jul 28 15:54:18.146052 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.146034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls\") pod \"a191d9c3-8d75-496b-867f-026862812283\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") "
Jul 28 15:54:18.146052 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.146054    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbvxr\" (UniqueName: \"kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr\") pod \"a191d9c3-8d75-496b-867f-026862812283\" (UID: \"a191d9c3-8d75-496b-867f-026862812283\") "
Jul 28 15:54:18.146359 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.146327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a191d9c3-8d75-496b-867f-026862812283" (UID: "a191d9c3-8d75-496b-867f-026862812283"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:54:18.148176 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.148149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr" (OuterVolumeSpecName: "kube-api-access-xbvxr") pod "a191d9c3-8d75-496b-867f-026862812283" (UID: "a191d9c3-8d75-496b-867f-026862812283"). InnerVolumeSpecName "kube-api-access-xbvxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:54:18.148296 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.148194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a191d9c3-8d75-496b-867f-026862812283" (UID: "a191d9c3-8d75-496b-867f-026862812283"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:54:18.246843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.246807    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a191d9c3-8d75-496b-867f-026862812283-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:18.246843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.246839    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a191d9c3-8d75-496b-867f-026862812283-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:18.246843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.246849    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvxr\" (UniqueName: \"kubernetes.io/projected/a191d9c3-8d75-496b-867f-026862812283-kube-api-access-xbvxr\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:18.363186 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.363151    2577 generic.go:356] "Generic (PLEG): container finished" podID="a191d9c3-8d75-496b-867f-026862812283" containerID="7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41" exitCode=0
Jul 28 15:54:18.363599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.363195    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"
Jul 28 15:54:18.363599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.363239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerDied","Data":"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"}
Jul 28 15:54:18.363599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.363303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8" event={"ID":"a191d9c3-8d75-496b-867f-026862812283","Type":"ContainerDied","Data":"860eebc06234c0bb431c8e950868f452185a4a68eafdd8ae2c6e8ff4328973b0"}
Jul 28 15:54:18.363599 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.363332    2577 scope.go:117] "RemoveContainer" containerID="aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"
Jul 28 15:54:18.377624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.377605    2577 scope.go:117] "RemoveContainer" containerID="7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"
Jul 28 15:54:18.385139 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.385121    2577 scope.go:117] "RemoveContainer" containerID="aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"
Jul 28 15:54:18.385409 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:18.385385    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10\": container with ID starting with aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10 not found: ID does not exist" containerID="aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"
Jul 28 15:54:18.385461 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.385417    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10"} err="failed to get container status \"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10\": rpc error: code = NotFound desc = could not find container \"aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10\": container with ID starting with aba56184e2bc9f8c61fd6af5c30062359aba2a32a533e9672ce8129830d8cc10 not found: ID does not exist"
Jul 28 15:54:18.385461 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.385436    2577 scope.go:117] "RemoveContainer" containerID="7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"
Jul 28 15:54:18.385687 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:18.385668    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41\": container with ID starting with 7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41 not found: ID does not exist" containerID="7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"
Jul 28 15:54:18.385727 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.385692    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41"} err="failed to get container status \"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41\": rpc error: code = NotFound desc = could not find container \"7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41\": container with ID starting with 7293402c5a728313f066caebb8c52ac8fb8d97340128217b87732677c4b1aa41 not found: ID does not exist"
Jul 28 15:54:18.391702 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.391675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:54:18.401320 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.401272    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b277-predictor-6f9fc5d8b6-5d5k8"]
Jul 28 15:54:18.561620 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:18.561588    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a191d9c3-8d75-496b-867f-026862812283" path="/var/lib/kubelet/pods/a191d9c3-8d75-496b-867f-026862812283/volumes"
Jul 28 15:54:22.362384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:22.362357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:54:22.362833 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:22.362790    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:22.533185 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:22.533139    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:27.532916 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:27.532864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:27.533388 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:27.533007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:54:32.363479 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:32.363438    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:32.533624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:32.533584    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:37.533386 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:37.533343    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:42.363491 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:42.363444    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:42.533334 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:42.533294    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:44.337820 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.337753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:54:44.338300 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.338002    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" containerID="cri-o://4947c48a45c748f2a91dd398bfa8287561c9c4e05dee17ec3234320b3e0e2660" gracePeriod=30
Jul 28 15:54:44.377835 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.377804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:44.411482 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.411451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:54:44.439128 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.439067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") pod \"43e5bece-01b6-40cf-a368-c1278d0afa4e\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") "
Jul 28 15:54:44.439293 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.439148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle\") pod \"43e5bece-01b6-40cf-a368-c1278d0afa4e\" (UID: \"43e5bece-01b6-40cf-a368-c1278d0afa4e\") "
Jul 28 15:54:44.439469 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.439447    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "43e5bece-01b6-40cf-a368-c1278d0afa4e" (UID: "43e5bece-01b6-40cf-a368-c1278d0afa4e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:54:44.441239 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.441211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43e5bece-01b6-40cf-a368-c1278d0afa4e" (UID: "43e5bece-01b6-40cf-a368-c1278d0afa4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:54:44.458177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.458143    2577 generic.go:356] "Generic (PLEG): container finished" podID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerID="5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab" exitCode=0
Jul 28 15:54:44.458312 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.458184    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"
Jul 28 15:54:44.458312 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.458219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" event={"ID":"43e5bece-01b6-40cf-a368-c1278d0afa4e","Type":"ContainerDied","Data":"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"}
Jul 28 15:54:44.458312 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.458276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h" event={"ID":"43e5bece-01b6-40cf-a368-c1278d0afa4e","Type":"ContainerDied","Data":"0e1ed74ef14ecc894ee526c1c376756f13b24f4b65bedb77382fe31b1f64a4d3"}
Jul 28 15:54:44.458312 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.458306    2577 scope.go:117] "RemoveContainer" containerID="5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"
Jul 28 15:54:44.468617 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.468599    2577 scope.go:117] "RemoveContainer" containerID="5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"
Jul 28 15:54:44.468921 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:44.468901    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab\": container with ID starting with 5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab not found: ID does not exist" containerID="5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"
Jul 28 15:54:44.468968 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.468935    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab"} err="failed to get container status \"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab\": rpc error: code = NotFound desc = could not find container \"5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab\": container with ID starting with 5232e87ce404b6f70fe14573d5d99ee2b1de04d033171333751b6c09fc696cab not found: ID does not exist"
Jul 28 15:54:44.517537 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.517439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:54:44.518768 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.518743    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4b277-54cd4bc6c7-w7b2h"]
Jul 28 15:54:44.540365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.540326    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e5bece-01b6-40cf-a368-c1278d0afa4e-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:44.540365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.540356    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43e5bece-01b6-40cf-a368-c1278d0afa4e-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:44.561841 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.561806    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" path="/var/lib/kubelet/pods/43e5bece-01b6-40cf-a368-c1278d0afa4e/volumes"
Jul 28 15:54:44.832311 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832233    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:54:44.832575 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832559    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container"
Jul 28 15:54:44.832632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832576    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container"
Jul 28 15:54:44.832632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832593    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kube-rbac-proxy"
Jul 28 15:54:44.832632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832599    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kube-rbac-proxy"
Jul 28 15:54:44.832632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832612    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277"
Jul 28 15:54:44.832632 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832618    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277"
Jul 28 15:54:44.832799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832665    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e5bece-01b6-40cf-a368-c1278d0afa4e" containerName="switch-graph-4b277"
Jul 28 15:54:44.832799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kserve-container"
Jul 28 15:54:44.832799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.832678    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a191d9c3-8d75-496b-867f-026862812283" containerName="kube-rbac-proxy"
Jul 28 15:54:44.836815 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.836782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.840165 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.840138    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12202-predictor-serving-cert\""
Jul 28 15:54:44.840294 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.840166    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12202-kube-rbac-proxy-sar-config\""
Jul 28 15:54:44.841944 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.841915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fv5\" (UniqueName: \"kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.842074 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.841959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.842074 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.842023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.868218 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.868192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:54:44.942917 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.942881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fv5\" (UniqueName: \"kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.943133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.942934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.943133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.942972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.943133 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:44.943058    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-12202-predictor-serving-cert: secret "error-404-isvc-12202-predictor-serving-cert" not found
Jul 28 15:54:44.943133 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:44.943126    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls podName:8deeb7a0-ec33-48f3-9bef-23fd4db43fb0 nodeName:}" failed. No retries permitted until 2026-07-28 15:54:45.443089141 +0000 UTC m=+1313.443329044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls") pod "error-404-isvc-12202-predictor-7fdc67944d-8cndv" (UID: "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0") : secret "error-404-isvc-12202-predictor-serving-cert" not found
Jul 28 15:54:44.943670 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.943637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:44.957257 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:44.957228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fv5\" (UniqueName: \"kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:45.045404 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.045369    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:54:45.045654 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.045629    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" containerID="cri-o://fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20" gracePeriod=30
Jul 28 15:54:45.045747 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.045656    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kube-rbac-proxy" containerID="cri-o://59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce" gracePeriod=30
Jul 28 15:54:45.445857 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.445767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:45.448304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.448277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") pod \"error-404-isvc-12202-predictor-7fdc67944d-8cndv\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") " pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:45.463777 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.463749    2577 generic.go:356] "Generic (PLEG): container finished" podID="babee5aa-d467-4143-892e-bbd46556aa8e" containerID="59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce" exitCode=2
Jul 28 15:54:45.463882 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.463822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerDied","Data":"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"}
Jul 28 15:54:45.747164 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.747132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:45.883648 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:45.883485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:54:45.885965 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:54:45.885936    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deeb7a0_ec33_48f3_9bef_23fd4db43fb0.slice/crio-3f594868518d0a6638c1f151d9c2035676da9cc72b7fd0517f7256cc1d4625e4 WatchSource:0}: Error finding container 3f594868518d0a6638c1f151d9c2035676da9cc72b7fd0517f7256cc1d4625e4: Status 404 returned error can't find the container with id 3f594868518d0a6638c1f151d9c2035676da9cc72b7fd0517f7256cc1d4625e4
Jul 28 15:54:46.065669 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.065637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:54:46.470515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.470428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerStarted","Data":"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"}
Jul 28 15:54:46.470515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.470496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerStarted","Data":"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"}
Jul 28 15:54:46.471019 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.470532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:46.471019 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.470559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerStarted","Data":"3f594868518d0a6638c1f151d9c2035676da9cc72b7fd0517f7256cc1d4625e4"}
Jul 28 15:54:46.500336 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.500264    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podStartSLOduration=2.500248026 podStartE2EDuration="2.500248026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:54:46.497668263 +0000 UTC m=+1314.497908187" watchObservedRunningTime="2026-07-28 15:54:46.500248026 +0000 UTC m=+1314.500487951"
Jul 28 15:54:46.680888 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:46.680850    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:47.452116 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:47.452049    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 15:54:47.457464 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:47.457436    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 15:54:47.475123 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:47.475084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:47.476344 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:47.476318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:54:48.296565 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.296539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:54:48.365961 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.365927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") pod \"babee5aa-d467-4143-892e-bbd46556aa8e\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") "
Jul 28 15:54:48.366124 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.365973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config\") pod \"babee5aa-d467-4143-892e-bbd46556aa8e\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") "
Jul 28 15:54:48.366364 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.366337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "babee5aa-d467-4143-892e-bbd46556aa8e" (UID: "babee5aa-d467-4143-892e-bbd46556aa8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:54:48.367981 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.367964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "babee5aa-d467-4143-892e-bbd46556aa8e" (UID: "babee5aa-d467-4143-892e-bbd46556aa8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:54:48.466662 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.466581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgrl7\" (UniqueName: \"kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7\") pod \"babee5aa-d467-4143-892e-bbd46556aa8e\" (UID: \"babee5aa-d467-4143-892e-bbd46556aa8e\") "
Jul 28 15:54:48.466799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.466749    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babee5aa-d467-4143-892e-bbd46556aa8e-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:48.466799 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.466761    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babee5aa-d467-4143-892e-bbd46556aa8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:48.468602 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.468579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7" (OuterVolumeSpecName: "kube-api-access-sgrl7") pod "babee5aa-d467-4143-892e-bbd46556aa8e" (UID: "babee5aa-d467-4143-892e-bbd46556aa8e"). InnerVolumeSpecName "kube-api-access-sgrl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:54:48.480178 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480153    2577 generic.go:356] "Generic (PLEG): container finished" podID="babee5aa-d467-4143-892e-bbd46556aa8e" containerID="fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20" exitCode=0
Jul 28 15:54:48.480495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480209    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"
Jul 28 15:54:48.480495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerDied","Data":"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"}
Jul 28 15:54:48.480495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw" event={"ID":"babee5aa-d467-4143-892e-bbd46556aa8e","Type":"ContainerDied","Data":"c03e551e347b361604287bdbdd778bd61d220e7e02005154ec39e1f8f8cfb75c"}
Jul 28 15:54:48.480495 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480320    2577 scope.go:117] "RemoveContainer" containerID="59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"
Jul 28 15:54:48.480731 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.480651    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:54:48.488405 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.488391    2577 scope.go:117] "RemoveContainer" containerID="fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"
Jul 28 15:54:48.495129 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.495114    2577 scope.go:117] "RemoveContainer" containerID="59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"
Jul 28 15:54:48.495362 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:48.495343    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce\": container with ID starting with 59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce not found: ID does not exist" containerID="59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"
Jul 28 15:54:48.495423 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.495372    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce"} err="failed to get container status \"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce\": rpc error: code = NotFound desc = could not find container \"59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce\": container with ID starting with 59bf994ceddfc0c7ddbefb9a3dc3e6d8b19aa1730fcc1fb5cf571f187cdf0dce not found: ID does not exist"
Jul 28 15:54:48.495423 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.495388    2577 scope.go:117] "RemoveContainer" containerID="fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"
Jul 28 15:54:48.495618 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:54:48.495599    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20\": container with ID starting with fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20 not found: ID does not exist" containerID="fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"
Jul 28 15:54:48.495663 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.495623    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20"} err="failed to get container status \"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20\": rpc error: code = NotFound desc = could not find container \"fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20\": container with ID starting with fa06ad425a4399a7882570d3964434881f94ab860d83f27136cd2e03b8aeeb20 not found: ID does not exist"
Jul 28 15:54:48.511340 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.511316    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:54:48.520019 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.519998    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-642bb-predictor-6c6846787d-fmpcw"]
Jul 28 15:54:48.561289 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.561262    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" path="/var/lib/kubelet/pods/babee5aa-d467-4143-892e-bbd46556aa8e/volumes"
Jul 28 15:54:48.567896 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:48.567877    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrl7\" (UniqueName: \"kubernetes.io/projected/babee5aa-d467-4143-892e-bbd46556aa8e-kube-api-access-sgrl7\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:54:51.680983 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:51.680944    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:52.363803 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:52.363764    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 15:54:53.486073 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:53.486037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:54:53.486571 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:53.486484    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:54:56.681195 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:56.681143    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:54:56.681659 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:54:56.681250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:55:01.680593 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:01.680546    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:02.364244 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:02.364215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:55:03.486629 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:03.486595    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:55:06.680643 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:06.680601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:11.681121 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:11.681061    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:13.486744 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:13.486704    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:55:14.585203 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.585163    2577 generic.go:356] "Generic (PLEG): container finished" podID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerID="4947c48a45c748f2a91dd398bfa8287561c9c4e05dee17ec3234320b3e0e2660" exitCode=0
Jul 28 15:55:14.585549 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.585222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" event={"ID":"180c1216-cc9b-4404-afc1-9d206f5bbeb2","Type":"ContainerDied","Data":"4947c48a45c748f2a91dd398bfa8287561c9c4e05dee17ec3234320b3e0e2660"}
Jul 28 15:55:14.637323 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.637232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:14.652194 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:14.652578 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652560    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container"
Jul 28 15:55:14.652578 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652580    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container"
Jul 28 15:55:14.652750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652598    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kube-rbac-proxy"
Jul 28 15:55:14.652750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652606    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kube-rbac-proxy"
Jul 28 15:55:14.652750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652676    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kube-rbac-proxy"
Jul 28 15:55:14.652750 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.652703    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="babee5aa-d467-4143-892e-bbd46556aa8e" containerName="kserve-container"
Jul 28 15:55:14.655770 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.655751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.659301 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.659278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ad35d-kube-rbac-proxy-sar-config\""
Jul 28 15:55:14.659389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.659310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ad35d-serving-cert\""
Jul 28 15:55:14.678369 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.678339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:14.747418 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.747388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.747563 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.747432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.848032 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.847998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.848236 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.848059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.848236 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:14.848186    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-ad35d-serving-cert: secret "ensemble-graph-ad35d-serving-cert" not found
Jul 28 15:55:14.848355 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:14.848267    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls podName:9faa982e-7304-4a68-ba08-d8d305c1888d nodeName:}" failed. No retries permitted until 2026-07-28 15:55:15.348242165 +0000 UTC m=+1343.348482087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls") pod "ensemble-graph-ad35d-7f5996df4c-kkpxm" (UID: "9faa982e-7304-4a68-ba08-d8d305c1888d") : secret "ensemble-graph-ad35d-serving-cert" not found
Jul 28 15:55:14.848772 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.848749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:14.942716 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.942685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:14.985657 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:14.985633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:55:15.049301 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.049274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") pod \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") "
Jul 28 15:55:15.049468 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.049338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle\") pod \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\" (UID: \"180c1216-cc9b-4404-afc1-9d206f5bbeb2\") "
Jul 28 15:55:15.049674 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.049653    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "180c1216-cc9b-4404-afc1-9d206f5bbeb2" (UID: "180c1216-cc9b-4404-afc1-9d206f5bbeb2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:15.051240 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.051210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "180c1216-cc9b-4404-afc1-9d206f5bbeb2" (UID: "180c1216-cc9b-4404-afc1-9d206f5bbeb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:15.150054 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.150021    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/180c1216-cc9b-4404-afc1-9d206f5bbeb2-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:15.150054 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.150048    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/180c1216-cc9b-4404-afc1-9d206f5bbeb2-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:15.351759 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.351721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:15.354075 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.354044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") pod \"ensemble-graph-ad35d-7f5996df4c-kkpxm\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") " pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:15.564752 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.564706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:15.590646 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.590607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh" event={"ID":"180c1216-cc9b-4404-afc1-9d206f5bbeb2","Type":"ContainerDied","Data":"0d9663900cd66ce3f33c86e577955152a4d32a780d767c156c73fae0a129c0ee"}
Jul 28 15:55:15.591251 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.590671    2577 scope.go:117] "RemoveContainer" containerID="4947c48a45c748f2a91dd398bfa8287561c9c4e05dee17ec3234320b3e0e2660"
Jul 28 15:55:15.591251 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.590621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"
Jul 28 15:55:15.618678 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.618647    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:55:15.629452 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.629426    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-642bb-65f5dc7d74-5xcjh"]
Jul 28 15:55:15.705517 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:15.705479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:15.708950 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:55:15.708914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faa982e_7304_4a68_ba08_d8d305c1888d.slice/crio-72f4422d563dceae27150612b5e1c310226f278d5b51eddb4e6fa844149993b9 WatchSource:0}: Error finding container 72f4422d563dceae27150612b5e1c310226f278d5b51eddb4e6fa844149993b9: Status 404 returned error can't find the container with id 72f4422d563dceae27150612b5e1c310226f278d5b51eddb4e6fa844149993b9
Jul 28 15:55:16.561778 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:16.561732    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" path="/var/lib/kubelet/pods/180c1216-cc9b-4404-afc1-9d206f5bbeb2/volumes"
Jul 28 15:55:16.596769 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:16.596734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" event={"ID":"9faa982e-7304-4a68-ba08-d8d305c1888d","Type":"ContainerStarted","Data":"ede8f91e24abfbcac9201587dbc9af9cc8face689e1623852f462644f66bbfa0"}
Jul 28 15:55:16.597281 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:16.596783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" event={"ID":"9faa982e-7304-4a68-ba08-d8d305c1888d","Type":"ContainerStarted","Data":"72f4422d563dceae27150612b5e1c310226f278d5b51eddb4e6fa844149993b9"}
Jul 28 15:55:16.597281 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:16.596879    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:16.620640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:16.620589    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podStartSLOduration=2.620576127 podStartE2EDuration="2.620576127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:16.619802103 +0000 UTC m=+1344.620042040" watchObservedRunningTime="2026-07-28 15:55:16.620576127 +0000 UTC m=+1344.620816135"
Jul 28 15:55:22.608535 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:22.608495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:23.486926 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:23.486889    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 15:55:24.706029 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.705991    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:24.706607 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.706226    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" containerID="cri-o://ede8f91e24abfbcac9201587dbc9af9cc8face689e1623852f462644f66bbfa0" gracePeriod=30
Jul 28 15:55:24.745528 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.745494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:24.874461 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.874419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:55:24.874755 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.874711    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container" containerID="cri-o://8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621" gracePeriod=30
Jul 28 15:55:24.874887 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.874747    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kube-rbac-proxy" containerID="cri-o://c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0" gracePeriod=30
Jul 28 15:55:24.937728 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:24.937688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:25.121728 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.121693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:25.199202 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.199168    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 15:55:25.199499 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.199484    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb"
Jul 28 15:55:25.199549 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.199501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb"
Jul 28 15:55:25.199604 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.199582    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="180c1216-cc9b-4404-afc1-9d206f5bbeb2" containerName="sequence-graph-642bb"
Jul 28 15:55:25.202513 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.202495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.205793 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.205771    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a87c3-predictor-serving-cert\""
Jul 28 15:55:25.206642 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.206618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a87c3-kube-rbac-proxy-sar-config\""
Jul 28 15:55:25.219251 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.219227    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 15:55:25.224515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.224482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.224636 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.224551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.224636 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.224599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbch\" (UniqueName: \"kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.266515 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.266425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:25.325332 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.325298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.325523 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.325352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.325523 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.325386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbch\" (UniqueName: \"kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.325975 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.325949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.327660 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.327642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.340695 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.340659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbch\" (UniqueName: \"kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch\") pod \"error-404-isvc-a87c3-predictor-578565c866-hnjq7\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") " pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.428611 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.428577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:25.512167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.512131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:25.593592 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.586848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:25.634624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.634586    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerID="c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0" exitCode=2
Jul 28 15:55:25.634624 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.634633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerDied","Data":"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"}
Jul 28 15:55:25.674713 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:55:25.674679    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2b6fe3_5b72_497a_860e_a7a19ecbcfa4.slice/crio-7007637acc16c3d7afa0d67e446e6ef66968e7cd97a03e9accb400b111cb495e WatchSource:0}: Error finding container 7007637acc16c3d7afa0d67e446e6ef66968e7cd97a03e9accb400b111cb495e: Status 404 returned error can't find the container with id 7007637acc16c3d7afa0d67e446e6ef66968e7cd97a03e9accb400b111cb495e
Jul 28 15:55:25.674948 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:25.674922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 15:55:26.641686 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:26.641643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerStarted","Data":"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"}
Jul 28 15:55:26.642133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:26.641723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerStarted","Data":"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"}
Jul 28 15:55:26.642133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:26.641754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerStarted","Data":"7007637acc16c3d7afa0d67e446e6ef66968e7cd97a03e9accb400b111cb495e"}
Jul 28 15:55:26.642133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:26.641875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:26.691216 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:26.691158    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podStartSLOduration=1.691142908 podStartE2EDuration="1.691142908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:26.69008984 +0000 UTC m=+1354.690329764" watchObservedRunningTime="2026-07-28 15:55:26.691142908 +0000 UTC m=+1354.691382833"
Jul 28 15:55:27.358503 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:27.358455    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 15:55:27.606958 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:27.606916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:27.645764 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:27.645681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:27.646814 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:27.646787    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:55:28.124348 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.124325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:55:28.250633 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.250600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x76j\" (UniqueName: \"kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j\") pod \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") "
Jul 28 15:55:28.250807 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.250667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls\") pod \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") "
Jul 28 15:55:28.250807 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.250738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config\") pod \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\" (UID: \"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2\") "
Jul 28 15:55:28.251112 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.251073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" (UID: "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:28.253133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.253088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j" (OuterVolumeSpecName: "kube-api-access-7x76j") pod "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" (UID: "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2"). InnerVolumeSpecName "kube-api-access-7x76j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:55:28.253133 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.253109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" (UID: "1d5cb8ed-0736-43af-b3b4-07b38ab90dc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:28.352149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.352090    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.352149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.352145    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.352149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.352155    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x76j\" (UniqueName: \"kubernetes.io/projected/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2-kube-api-access-7x76j\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.650694 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.650662    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerID="8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621" exitCode=0
Jul 28 15:55:28.651157 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.650714    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"
Jul 28 15:55:28.651157 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.650747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerDied","Data":"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"}
Jul 28 15:55:28.651157 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.650829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh" event={"ID":"1d5cb8ed-0736-43af-b3b4-07b38ab90dc2","Type":"ContainerDied","Data":"342cc8759b7c495f96119a12304ba031dd955f5ba8758bac932e35bf22a24a45"}
Jul 28 15:55:28.651157 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.650871    2577 scope.go:117] "RemoveContainer" containerID="c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"
Jul 28 15:55:28.651469 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.651440    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:55:28.659175 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.659156    2577 scope.go:117] "RemoveContainer" containerID="8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"
Jul 28 15:55:28.666675 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.666648    2577 scope.go:117] "RemoveContainer" containerID="c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"
Jul 28 15:55:28.666933 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:28.666910    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0\": container with ID starting with c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0 not found: ID does not exist" containerID="c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"
Jul 28 15:55:28.666992 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.666943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0"} err="failed to get container status \"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0\": rpc error: code = NotFound desc = could not find container \"c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0\": container with ID starting with c68f9f889df00c315d404df70548b71fc74ef13944f50a1e111dc9dbaf677cb0 not found: ID does not exist"
Jul 28 15:55:28.666992 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.666960    2577 scope.go:117] "RemoveContainer" containerID="8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"
Jul 28 15:55:28.667244 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:28.667225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621\": container with ID starting with 8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621 not found: ID does not exist" containerID="8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"
Jul 28 15:55:28.667306 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.667250    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621"} err="failed to get container status \"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621\": rpc error: code = NotFound desc = could not find container \"8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621\": container with ID starting with 8b575cc540e3a556139fcd3c8f933e304dc89a073544765782a31dfcfff0b621 not found: ID does not exist"
Jul 28 15:55:28.681805 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.681774    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:55:28.691109 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:28.691070    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad35d-predictor-6cb9647b76-schlh"]
Jul 28 15:55:30.560848 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:30.560803    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" path="/var/lib/kubelet/pods/1d5cb8ed-0736-43af-b3b4-07b38ab90dc2/volumes"
Jul 28 15:55:32.605865 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:32.605826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:33.487968 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:33.487937    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:55:33.655925 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:33.655887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:55:33.656399 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:33.656373    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:55:37.607072 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:37.607031    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:37.607564 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:37.607163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:42.606007 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:42.605962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:43.657320 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:43.657278    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:55:44.597496 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597438    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:55:44.597796 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597781    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kube-rbac-proxy"
Jul 28 15:55:44.597843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597798    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kube-rbac-proxy"
Jul 28 15:55:44.597843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container"
Jul 28 15:55:44.597843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597818    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container"
Jul 28 15:55:44.597947 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597872    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kserve-container"
Jul 28 15:55:44.597947 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.597883    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5cb8ed-0736-43af-b3b4-07b38ab90dc2" containerName="kube-rbac-proxy"
Jul 28 15:55:44.602041 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.602021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.605420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.605394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-12202-kube-rbac-proxy-sar-config\""
Jul 28 15:55:44.605420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.605400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-12202-serving-cert\""
Jul 28 15:55:44.617567 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.617539    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:55:44.627756 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.627727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:44.685657 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.685610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.686043 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.685691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.739396 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.739345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:44.786736 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.786697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.786929 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.786753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.787493 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.787468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.789147 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.789118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls\") pod \"sequence-graph-12202-954b76fdc-jkkll\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") " pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:44.912912 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:44.912820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:45.049935 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:45.049909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:55:45.052046 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:55:45.052015    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbdb8d0_f4cc_42d9_988c_b741fa57502b.slice/crio-8093f4bc1feafe7e2be26f44a2ffe09540cdcd743af019dae1f98c5e3109dd62 WatchSource:0}: Error finding container 8093f4bc1feafe7e2be26f44a2ffe09540cdcd743af019dae1f98c5e3109dd62: Status 404 returned error can't find the container with id 8093f4bc1feafe7e2be26f44a2ffe09540cdcd743af019dae1f98c5e3109dd62
Jul 28 15:55:45.717346 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:45.717311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" event={"ID":"5fbdb8d0-f4cc-42d9-988c-b741fa57502b","Type":"ContainerStarted","Data":"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"}
Jul 28 15:55:45.717822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:45.717364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" event={"ID":"5fbdb8d0-f4cc-42d9-988c-b741fa57502b","Type":"ContainerStarted","Data":"8093f4bc1feafe7e2be26f44a2ffe09540cdcd743af019dae1f98c5e3109dd62"}
Jul 28 15:55:45.717822 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:45.717393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:45.740603 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:45.740541    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podStartSLOduration=1.740519833 podStartE2EDuration="1.740519833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:45.740277248 +0000 UTC m=+1373.740517173" watchObservedRunningTime="2026-07-28 15:55:45.740519833 +0000 UTC m=+1373.740759762"
Jul 28 15:55:47.606259 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:47.606218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:51.726851 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:51.726821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:55:52.606692 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:52.606649    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:53.656347 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:53.656299    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:55:54.647286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.647252    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:55:54.647566 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.647446    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" containerID="cri-o://ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061" gracePeriod=30
Jul 28 15:55:54.686888 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.686861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:54.753654 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.753620    2577 generic.go:356] "Generic (PLEG): container finished" podID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerID="ede8f91e24abfbcac9201587dbc9af9cc8face689e1623852f462644f66bbfa0" exitCode=0
Jul 28 15:55:54.753798 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.753685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" event={"ID":"9faa982e-7304-4a68-ba08-d8d305c1888d","Type":"ContainerDied","Data":"ede8f91e24abfbcac9201587dbc9af9cc8face689e1623852f462644f66bbfa0"}
Jul 28 15:55:54.848728 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.848699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:54.872024 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.871996    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:54.933290 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.933219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:55:54.933456 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.933430    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container" containerID="cri-o://78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52" gracePeriod=30
Jul 28 15:55:54.933498 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.933463    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kube-rbac-proxy" containerID="cri-o://9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d" gracePeriod=30
Jul 28 15:55:54.966422 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.966396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:54.967376 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.967357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle\") pod \"9faa982e-7304-4a68-ba08-d8d305c1888d\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") "
Jul 28 15:55:54.967434 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.967423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") pod \"9faa982e-7304-4a68-ba08-d8d305c1888d\" (UID: \"9faa982e-7304-4a68-ba08-d8d305c1888d\") "
Jul 28 15:55:54.967653 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.967636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9faa982e-7304-4a68-ba08-d8d305c1888d" (UID: "9faa982e-7304-4a68-ba08-d8d305c1888d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:54.969490 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:54.969460    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9faa982e-7304-4a68-ba08-d8d305c1888d" (UID: "9faa982e-7304-4a68-ba08-d8d305c1888d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:55.060770 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.060738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:55.068140 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.068118    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa982e-7304-4a68-ba08-d8d305c1888d-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:55.068140 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.068139    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faa982e-7304-4a68-ba08-d8d305c1888d-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:55.082177 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.082149    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 15:55:55.082445 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.082433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d"
Jul 28 15:55:55.082484 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.082447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d"
Jul 28 15:55:55.082526 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.082515    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" containerName="ensemble-graph-ad35d"
Jul 28 15:55:55.085352 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.085337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.088286 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.088269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-50d42-predictor-serving-cert\""
Jul 28 15:55:55.088370 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.088290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-50d42-kube-rbac-proxy-sar-config\""
Jul 28 15:55:55.101965 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.101944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 15:55:55.169389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.169349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.169548 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.169400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.169548 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.169430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6blg\" (UniqueName: \"kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.269898 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.269857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.270142 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.269917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.270142 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.269947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6blg\" (UniqueName: \"kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.270142 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:55.270085    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-50d42-predictor-serving-cert: secret "error-404-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.270284 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:55.270176    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls podName:529d4acb-1ed1-46d2-a4cc-e89c1982008a nodeName:}" failed. No retries permitted until 2026-07-28 15:55:55.770155963 +0000 UTC m=+1383.770395878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls") pod "error-404-isvc-50d42-predictor-8b88c967b-q9295" (UID: "529d4acb-1ed1-46d2-a4cc-e89c1982008a") : secret "error-404-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.270572 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.270552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.285887 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.285854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6blg\" (UniqueName: \"kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.351510 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.351478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:55.759084 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.759053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm" event={"ID":"9faa982e-7304-4a68-ba08-d8d305c1888d","Type":"ContainerDied","Data":"72f4422d563dceae27150612b5e1c310226f278d5b51eddb4e6fa844149993b9"}
Jul 28 15:55:55.759579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.759135    2577 scope.go:117] "RemoveContainer" containerID="ede8f91e24abfbcac9201587dbc9af9cc8face689e1623852f462644f66bbfa0"
Jul 28 15:55:55.759579 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.759063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"
Jul 28 15:55:55.761167 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.761139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerDied","Data":"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"}
Jul 28 15:55:55.761332 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.761089    2577 generic.go:356] "Generic (PLEG): container finished" podID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerID="9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d" exitCode=2
Jul 28 15:55:55.773678 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.773655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:55.773800 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:55.773782    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-50d42-predictor-serving-cert: secret "error-404-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.773880 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:55.773870    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls podName:529d4acb-1ed1-46d2-a4cc-e89c1982008a nodeName:}" failed. No retries permitted until 2026-07-28 15:55:56.773856442 +0000 UTC m=+1384.774096344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls") pod "error-404-isvc-50d42-predictor-8b88c967b-q9295" (UID: "529d4acb-1ed1-46d2-a4cc-e89c1982008a") : secret "error-404-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.785033 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.784995    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:55.791189 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:55.791165    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ad35d-7f5996df4c-kkpxm"]
Jul 28 15:55:56.004647 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.004609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:56.561425 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.561393    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faa982e-7304-4a68-ba08-d8d305c1888d" path="/var/lib/kubelet/pods/9faa982e-7304-4a68-ba08-d8d305c1888d/volumes"
Jul 28 15:55:56.725666 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.725624    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:55:56.783713 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.783672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:56.786242 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.786216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"error-404-isvc-50d42-predictor-8b88c967b-q9295\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") " pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:56.895306 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:56.895205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:57.033800 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.033776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 15:55:57.036449 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:55:57.036418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529d4acb_1ed1_46d2_a4cc_e89c1982008a.slice/crio-4881466b27360c5c04f1d46d2e50a791f5bde0f46017f211c505197b8a2915c5 WatchSource:0}: Error finding container 4881466b27360c5c04f1d46d2e50a791f5bde0f46017f211c505197b8a2915c5: Status 404 returned error can't find the container with id 4881466b27360c5c04f1d46d2e50a791f5bde0f46017f211c505197b8a2915c5
Jul 28 15:55:57.771747 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.771708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerStarted","Data":"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"}
Jul 28 15:55:57.771945 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.771770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerStarted","Data":"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"}
Jul 28 15:55:57.771945 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.771789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerStarted","Data":"4881466b27360c5c04f1d46d2e50a791f5bde0f46017f211c505197b8a2915c5"}
Jul 28 15:55:57.771945 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.771877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:57.792792 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:57.792728    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podStartSLOduration=2.792706773 podStartE2EDuration="2.792706773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:57.792166173 +0000 UTC m=+1385.792406119" watchObservedRunningTime="2026-07-28 15:55:57.792706773 +0000 UTC m=+1385.792946699"
Jul 28 15:55:58.481703 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.481664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 15:55:58.589140 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.589063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:55:58.700426 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.700382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config\") pod \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") "
Jul 28 15:55:58.700426 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.700433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5fv5\" (UniqueName: \"kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5\") pod \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") "
Jul 28 15:55:58.700644 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.700526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") pod \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\" (UID: \"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0\") "
Jul 28 15:55:58.700858 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.700829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" (UID: "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:58.702555 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.702536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" (UID: "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:58.702628 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.702614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5" (OuterVolumeSpecName: "kube-api-access-n5fv5") pod "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" (UID: "8deeb7a0-ec33-48f3-9bef-23fd4db43fb0"). InnerVolumeSpecName "kube-api-access-n5fv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:55:58.776404 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776366    2577 generic.go:356] "Generic (PLEG): container finished" podID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerID="78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52" exitCode=0
Jul 28 15:55:58.776588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776417    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"
Jul 28 15:55:58.776588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerDied","Data":"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"}
Jul 28 15:55:58.776588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv" event={"ID":"8deeb7a0-ec33-48f3-9bef-23fd4db43fb0","Type":"ContainerDied","Data":"3f594868518d0a6638c1f151d9c2035676da9cc72b7fd0517f7256cc1d4625e4"}
Jul 28 15:55:58.776588 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776536    2577 scope.go:117] "RemoveContainer" containerID="9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"
Jul 28 15:55:58.776893 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.776872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:55:58.778113 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.778073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:55:58.785043 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.785023    2577 scope.go:117] "RemoveContainer" containerID="78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"
Jul 28 15:55:58.792356 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.792338    2577 scope.go:117] "RemoveContainer" containerID="9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"
Jul 28 15:55:58.792610 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:58.792590    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d\": container with ID starting with 9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d not found: ID does not exist" containerID="9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"
Jul 28 15:55:58.792663 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.792618    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d"} err="failed to get container status \"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d\": rpc error: code = NotFound desc = could not find container \"9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d\": container with ID starting with 9a3672fb5aa555b0cba7c72ea79c960eed7b71fe69baa7df4f4441e1949ef74d not found: ID does not exist"
Jul 28 15:55:58.792663 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.792635    2577 scope.go:117] "RemoveContainer" containerID="78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"
Jul 28 15:55:58.792893 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:55:58.792874    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52\": container with ID starting with 78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52 not found: ID does not exist" containerID="78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"
Jul 28 15:55:58.793149 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.792897    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52"} err="failed to get container status \"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52\": rpc error: code = NotFound desc = could not find container \"78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52\": container with ID starting with 78a2b381440019eb8ff394506936cd302a997487dcdf42f0497da0e4f06e4c52 not found: ID does not exist"
Jul 28 15:55:58.801289 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.801265    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.801289 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.801286    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.801410 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.801297    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fv5\" (UniqueName: \"kubernetes.io/projected/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0-kube-api-access-n5fv5\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.804147 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.804126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:55:58.807749 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:58.807731    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12202-predictor-7fdc67944d-8cndv"]
Jul 28 15:55:59.781774 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:55:59.781730    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:56:00.561786 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:00.561754    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" path="/var/lib/kubelet/pods/8deeb7a0-ec33-48f3-9bef-23fd4db43fb0/volumes"
Jul 28 15:56:01.727824 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:01.727249    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:56:03.656669 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:03.656630    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 15:56:04.786928 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:04.786899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:56:04.787389 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:04.787350    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:56:06.725897 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:06.725856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:56:06.726350 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:06.725974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:56:11.725725 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:11.725676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:56:13.657255 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:13.657213    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 15:56:14.787667 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:14.787622    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:56:16.725338 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:16.725288    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:56:21.725871 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:21.725833    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:56:24.787774 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.787736    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:56:24.792613 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.792592    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:56:24.811839 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.811795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls\") pod \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") "
Jul 28 15:56:24.811998 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.811882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle\") pod \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\" (UID: \"5fbdb8d0-f4cc-42d9-988c-b741fa57502b\") "
Jul 28 15:56:24.812361 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.812318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5fbdb8d0-f4cc-42d9-988c-b741fa57502b" (UID: "5fbdb8d0-f4cc-42d9-988c-b741fa57502b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:56:24.814420 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.814385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fbdb8d0-f4cc-42d9-988c-b741fa57502b" (UID: "5fbdb8d0-f4cc-42d9-988c-b741fa57502b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:56:24.874345 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.874305    2577 generic.go:356] "Generic (PLEG): container finished" podID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerID="ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061" exitCode=0
Jul 28 15:56:24.874509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.874354    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"
Jul 28 15:56:24.874509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.874388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" event={"ID":"5fbdb8d0-f4cc-42d9-988c-b741fa57502b","Type":"ContainerDied","Data":"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"}
Jul 28 15:56:24.874509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.874441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll" event={"ID":"5fbdb8d0-f4cc-42d9-988c-b741fa57502b","Type":"ContainerDied","Data":"8093f4bc1feafe7e2be26f44a2ffe09540cdcd743af019dae1f98c5e3109dd62"}
Jul 28 15:56:24.874509 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.874463    2577 scope.go:117] "RemoveContainer" containerID="ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"
Jul 28 15:56:24.882413 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.882385    2577 scope.go:117] "RemoveContainer" containerID="ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"
Jul 28 15:56:24.882717 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:56:24.882695    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061\": container with ID starting with ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061 not found: ID does not exist" containerID="ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"
Jul 28 15:56:24.882807 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.882730    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061"} err="failed to get container status \"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061\": rpc error: code = NotFound desc = could not find container \"ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061\": container with ID starting with ae3520d7863d7c507f576f5ebf777d26c432699ad53938a5603604dbd3563061 not found: ID does not exist"
Jul 28 15:56:24.899479 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.899454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:56:24.908058 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.908001    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12202-954b76fdc-jkkll"]
Jul 28 15:56:24.913030 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.913012    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:56:24.913121 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.913035    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbdb8d0-f4cc-42d9-988c-b741fa57502b-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 15:56:24.980055 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 15:56:24.980337 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container"
Jul 28 15:56:24.980384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container"
Jul 28 15:56:24.980384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202"
Jul 28 15:56:24.980384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980359    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202"
Jul 28 15:56:24.980384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kube-rbac-proxy"
Jul 28 15:56:24.980384 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kube-rbac-proxy"
Jul 28 15:56:24.980555 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980427    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" containerName="sequence-graph-12202"
Jul 28 15:56:24.980555 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980435    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kserve-container"
Jul 28 15:56:24.980555 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.980442    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8deeb7a0-ec33-48f3-9bef-23fd4db43fb0" containerName="kube-rbac-proxy"
Jul 28 15:56:24.984783 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.984763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:24.987835 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.987813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-a87c3-kube-rbac-proxy-sar-config\""
Jul 28 15:56:24.988205 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.988179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-a87c3-serving-cert\""
Jul 28 15:56:24.988811 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.988789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:24.995062 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:24.995039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 15:56:25.014371 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.014343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.014496 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.014395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.079161 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.079126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:25.115247 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.115215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.115414 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.115279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.115842 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.115822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.117544 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.117523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls\") pod \"ensemble-graph-a87c3-6c9b8648b7-bmf4f\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") " pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.295640 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.295601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.435328 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.435303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 15:56:25.437126 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:56:25.437075    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f2ec7d_e239_4761_8142_35c1e124d730.slice/crio-abf3cc84d2005acd319317fdc9fcad55f08980b58ab75bf4624c162e2190d741 WatchSource:0}: Error finding container abf3cc84d2005acd319317fdc9fcad55f08980b58ab75bf4624c162e2190d741: Status 404 returned error can't find the container with id abf3cc84d2005acd319317fdc9fcad55f08980b58ab75bf4624c162e2190d741
Jul 28 15:56:25.880922 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.880887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" event={"ID":"10f2ec7d-e239-4761-8142-35c1e124d730","Type":"ContainerStarted","Data":"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"}
Jul 28 15:56:25.881363 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.880940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" event={"ID":"10f2ec7d-e239-4761-8142-35c1e124d730","Type":"ContainerStarted","Data":"abf3cc84d2005acd319317fdc9fcad55f08980b58ab75bf4624c162e2190d741"}
Jul 28 15:56:25.881363 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.881021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:25.901233 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:25.901185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podStartSLOduration=1.9011674840000001 podStartE2EDuration="1.901167484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:56:25.899775996 +0000 UTC m=+1413.900015957" watchObservedRunningTime="2026-07-28 15:56:25.901167484 +0000 UTC m=+1413.901407408"
Jul 28 15:56:26.561345 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:26.561311    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbdb8d0-f4cc-42d9-988c-b741fa57502b" path="/var/lib/kubelet/pods/5fbdb8d0-f4cc-42d9-988c-b741fa57502b/volumes"
Jul 28 15:56:31.897304 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:31.897263    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 15:56:34.787603 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:34.787556    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 15:56:44.788297 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:44.788222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 15:56:54.949692 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.949655    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 15:56:54.954289 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.954269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:54.957128 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.957104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-50d42-serving-cert\""
Jul 28 15:56:54.957228 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.957088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-50d42-kube-rbac-proxy-sar-config\""
Jul 28 15:56:54.961366 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.960930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 15:56:54.981546 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:54.981513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:55.049224 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.049187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.049365 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.049293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.149898 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.149861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.150089 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.149927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.150089 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:56:55.150062    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-50d42-serving-cert: secret "sequence-graph-50d42-serving-cert" not found
Jul 28 15:56:55.150193 ip-10-0-134-118 kubenswrapper[2577]: E0728 15:56:55.150160    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls podName:3dafbfb3-c9a9-4c29-93c3-aac3353e2452 nodeName:}" failed. No retries permitted until 2026-07-28 15:56:55.65013826 +0000 UTC m=+1443.650378164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls") pod "sequence-graph-50d42-799cd94864-7vggx" (UID: "3dafbfb3-c9a9-4c29-93c3-aac3353e2452") : secret "sequence-graph-50d42-serving-cert" not found
Jul 28 15:56:55.150557 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.150536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.628262 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.628225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:55.654425 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.654383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.656798 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.656774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") pod \"sequence-graph-50d42-799cd94864-7vggx\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") " pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.868980 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.868931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:55.998656 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:55.998591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 15:56:56.001190 ip-10-0-134-118 kubenswrapper[2577]: W0728 15:56:56.001155    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dafbfb3_c9a9_4c29_93c3_aac3353e2452.slice/crio-c674b83c053b8611d65ab84d23605d79605e79c91f1388616e5a74e6020a610e WatchSource:0}: Error finding container c674b83c053b8611d65ab84d23605d79605e79c91f1388616e5a74e6020a610e: Status 404 returned error can't find the container with id c674b83c053b8611d65ab84d23605d79605e79c91f1388616e5a74e6020a610e
Jul 28 15:56:57.002274 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:57.002240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" event={"ID":"3dafbfb3-c9a9-4c29-93c3-aac3353e2452","Type":"ContainerStarted","Data":"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"}
Jul 28 15:56:57.002691 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:57.002294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" event={"ID":"3dafbfb3-c9a9-4c29-93c3-aac3353e2452","Type":"ContainerStarted","Data":"c674b83c053b8611d65ab84d23605d79605e79c91f1388616e5a74e6020a610e"}
Jul 28 15:56:57.002691 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:57.002338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:56:57.019804 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:56:57.019754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podStartSLOduration=3.01973759 podStartE2EDuration="3.01973759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:56:57.018852031 +0000 UTC m=+1445.019091956" watchObservedRunningTime="2026-07-28 15:56:57.01973759 +0000 UTC m=+1445.019977514"
Jul 28 15:57:03.012343 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:57:03.012311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 15:57:33.164397 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:57:33.164361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:57:52.450417 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:57:52.450390    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:57:52.451236 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:57:52.451217    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 15:58:58.074820 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:58:58.074786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:08.559843 ip-10-0-134-118 kubenswrapper[2577]: I0728 15:59:08.559803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:50.533240 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:02:50.533206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:52.471470 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:02:52.471442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:02:52.473678 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:02:52.473657    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:04:28.788297 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:28.788263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:39.502280 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.502242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 16:04:39.503147 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.503072    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" containerID="cri-o://a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2" gracePeriod=30
Jul 28 16:04:39.526015 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.525983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:39.677187 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.677153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:39.767436 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.767339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:39.812440 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.812400    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:04:39.815798 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.815773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:39.818813 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.818788    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b87a7-predictor-serving-cert\""
Jul 28 16:04:39.818979 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.818945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b87a7-kube-rbac-proxy-sar-config\""
Jul 28 16:04:39.828365 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.828337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:04:39.854736 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.854643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:39.866695 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.866663    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 16:04:39.866948 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.866915    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container" containerID="cri-o://b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b" gracePeriod=30
Jul 28 16:04:39.866948 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.866939    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kube-rbac-proxy" containerID="cri-o://8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba" gracePeriod=30
Jul 28 16:04:39.911697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.911664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2xb\" (UniqueName: \"kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:39.911842 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.911710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:39.911842 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.911739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:39.921819 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:39.921792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:40.012996 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.012961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2xb\" (UniqueName: \"kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.013180 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.013009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.013180 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.013041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.013638 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.013618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.015549 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.015525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.025058 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.024997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2xb\" (UniqueName: \"kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb\") pod \"error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") " pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.125900 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.125854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.171248 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.171210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:40.261006 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.260977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:04:40.263322 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:04:40.263293    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a1bb7a_5770_4953_abb6_d23aa5ecf273.slice/crio-9e09096aba3af754cb96a1b9e60749b2ab2f9ce38789e53e9bff1277503f6737 WatchSource:0}: Error finding container 9e09096aba3af754cb96a1b9e60749b2ab2f9ce38789e53e9bff1277503f6737: Status 404 returned error can't find the container with id 9e09096aba3af754cb96a1b9e60749b2ab2f9ce38789e53e9bff1277503f6737
Jul 28 16:04:40.696739 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.696642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerStarted","Data":"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"}
Jul 28 16:04:40.696739 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.696721    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:40.697248 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.696745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerStarted","Data":"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"}
Jul 28 16:04:40.697248 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.696773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerStarted","Data":"9e09096aba3af754cb96a1b9e60749b2ab2f9ce38789e53e9bff1277503f6737"}
Jul 28 16:04:40.698479 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.698452    2577 generic.go:356] "Generic (PLEG): container finished" podID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerID="8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba" exitCode=2
Jul 28 16:04:40.698589 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.698521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerDied","Data":"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"}
Jul 28 16:04:40.718228 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:40.718174    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podStartSLOduration=1.7181541679999999 podStartE2EDuration="1.718154168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:40.717620622 +0000 UTC m=+1908.717860547" watchObservedRunningTime="2026-07-28 16:04:40.718154168 +0000 UTC m=+1908.718394092"
Jul 28 16:04:41.702370 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:41.702334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:41.703735 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:41.703703    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:04:41.895318 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:41.895278    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:04:42.706161 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:42.706124    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:04:43.117179 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.117157    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 16:04:43.139511 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.139479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls\") pod \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") "
Jul 28 16:04:43.139661 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.139551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config\") pod \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") "
Jul 28 16:04:43.139661 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.139597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwbch\" (UniqueName: \"kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch\") pod \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\" (UID: \"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4\") "
Jul 28 16:04:43.140027 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.139955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" (UID: "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:43.142138 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.142112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" (UID: "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:43.142259 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.142127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch" (OuterVolumeSpecName: "kube-api-access-cwbch") pod "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" (UID: "de2b6fe3-5b72-497a-860e-a7a19ecbcfa4"). InnerVolumeSpecName "kube-api-access-cwbch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:43.240698 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.240622    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.240698 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.240651    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.240698 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.240662    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbch\" (UniqueName: \"kubernetes.io/projected/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4-kube-api-access-cwbch\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.710706 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.710614    2577 generic.go:356] "Generic (PLEG): container finished" podID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerID="b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b" exitCode=0
Jul 28 16:04:43.710706 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.710660    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"
Jul 28 16:04:43.710706 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.710695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerDied","Data":"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"}
Jul 28 16:04:43.711253 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.710752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7" event={"ID":"de2b6fe3-5b72-497a-860e-a7a19ecbcfa4","Type":"ContainerDied","Data":"7007637acc16c3d7afa0d67e446e6ef66968e7cd97a03e9accb400b111cb495e"}
Jul 28 16:04:43.711253 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.710780    2577 scope.go:117] "RemoveContainer" containerID="8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"
Jul 28 16:04:43.719128 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.719087    2577 scope.go:117] "RemoveContainer" containerID="b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"
Jul 28 16:04:43.725970 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.725949    2577 scope.go:117] "RemoveContainer" containerID="8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"
Jul 28 16:04:43.726285 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:04:43.726266    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba\": container with ID starting with 8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba not found: ID does not exist" containerID="8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"
Jul 28 16:04:43.726362 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.726292    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba"} err="failed to get container status \"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba\": rpc error: code = NotFound desc = could not find container \"8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba\": container with ID starting with 8ff84d2003680da0c7c43eca8db5cbbc248d0b8f689a1045b623f46703e89fba not found: ID does not exist"
Jul 28 16:04:43.726362 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.726309    2577 scope.go:117] "RemoveContainer" containerID="b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"
Jul 28 16:04:43.726568 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:04:43.726548    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b\": container with ID starting with b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b not found: ID does not exist" containerID="b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"
Jul 28 16:04:43.726610 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.726574    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b"} err="failed to get container status \"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b\": rpc error: code = NotFound desc = could not find container \"b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b\": container with ID starting with b664201ed91c14c201532574010d717693a8d94550f15ddc9a2f26c29407ca4b not found: ID does not exist"
Jul 28 16:04:43.732079 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.732055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 16:04:43.735444 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:43.735424    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a87c3-predictor-578565c866-hnjq7"]
Jul 28 16:04:44.561278 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:44.561241    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" path="/var/lib/kubelet/pods/de2b6fe3-5b72-497a-860e-a7a19ecbcfa4/volumes"
Jul 28 16:04:46.894825 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:46.894780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:04:47.711986 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:47.711953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:04:47.712454 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:47.712418    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:04:51.894929 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:51.894887    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:04:51.895381 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:51.895000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 16:04:56.894332 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:56.894287    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:04:57.713150 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:04:57.713105    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:05:01.894518 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:01.894474    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:06.894593 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:06.894557    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:07.712739 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:07.712702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:05:09.603254 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.603187    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 16:05:09.604206 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.604159    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" containerID="cri-o://695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4" gracePeriod=30
Jul 28 16:05:09.626134 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.626050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:09.685227 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.685201    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 16:05:09.761428 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.761347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls\") pod \"10f2ec7d-e239-4761-8142-35c1e124d730\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") "
Jul 28 16:05:09.761428 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.761422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle\") pod \"10f2ec7d-e239-4761-8142-35c1e124d730\" (UID: \"10f2ec7d-e239-4761-8142-35c1e124d730\") "
Jul 28 16:05:09.761776 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.761751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "10f2ec7d-e239-4761-8142-35c1e124d730" (UID: "10f2ec7d-e239-4761-8142-35c1e124d730"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:09.763551 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.763519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10f2ec7d-e239-4761-8142-35c1e124d730" (UID: "10f2ec7d-e239-4761-8142-35c1e124d730"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:09.764217 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.764169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:09.796909 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.796876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 16:05:09.797147 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.797089    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container" containerID="cri-o://d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3" gracePeriod=30
Jul 28 16:05:09.797271 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.797157    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kube-rbac-proxy" containerID="cri-o://f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35" gracePeriod=30
Jul 28 16:05:09.814817 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.814789    2577 generic.go:356] "Generic (PLEG): container finished" podID="10f2ec7d-e239-4761-8142-35c1e124d730" containerID="a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2" exitCode=0
Jul 28 16:05:09.814950 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.814836    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"
Jul 28 16:05:09.814950 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.814870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" event={"ID":"10f2ec7d-e239-4761-8142-35c1e124d730","Type":"ContainerDied","Data":"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"}
Jul 28 16:05:09.814950 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.814937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f" event={"ID":"10f2ec7d-e239-4761-8142-35c1e124d730","Type":"ContainerDied","Data":"abf3cc84d2005acd319317fdc9fcad55f08980b58ab75bf4624c162e2190d741"}
Jul 28 16:05:09.815146 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.814967    2577 scope.go:117] "RemoveContainer" containerID="a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"
Jul 28 16:05:09.838946 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.838915    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 16:05:09.841114 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.841076    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a87c3-6c9b8648b7-bmf4f"]
Jul 28 16:05:09.862642 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.862574    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f2ec7d-e239-4761-8142-35c1e124d730-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:09.862642 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.862609    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f2ec7d-e239-4761-8142-35c1e124d730-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:09.863738 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.863707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:09.907940 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.907919    2577 scope.go:117] "RemoveContainer" containerID="a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"
Jul 28 16:05:09.908324 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:09.908297    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2\": container with ID starting with a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2 not found: ID does not exist" containerID="a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"
Jul 28 16:05:09.908419 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.908339    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2"} err="failed to get container status \"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2\": rpc error: code = NotFound desc = could not find container \"a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2\": container with ID starting with a8d18b6326db560f52e65459f1855d4eb8440d8025ac9392ef6515c493b006c2 not found: ID does not exist"
Jul 28 16:05:09.912151 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:05:09.912607 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container"
Jul 28 16:05:09.912679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912610    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container"
Jul 28 16:05:09.912679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912626    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3"
Jul 28 16:05:09.912679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912634    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3"
Jul 28 16:05:09.912679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912650    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kube-rbac-proxy"
Jul 28 16:05:09.912679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912658    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kube-rbac-proxy"
Jul 28 16:05:09.912870 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912733    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kserve-container"
Jul 28 16:05:09.912870 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912742    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2b6fe3-5b72-497a-860e-a7a19ecbcfa4" containerName="kube-rbac-proxy"
Jul 28 16:05:09.912870 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.912748    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" containerName="ensemble-graph-a87c3"
Jul 28 16:05:09.915645 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.915627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:09.918253 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.918227    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bd785-predictor-serving-cert\""
Jul 28 16:05:09.918413 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.918397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bd785-kube-rbac-proxy-sar-config\""
Jul 28 16:05:09.925089 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.925059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:05:09.961971 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:09.961937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:10.064364 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.064261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.064364 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.064312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.064630 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.064430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbfs\" (UniqueName: \"kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.145860 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.145830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:10.165721 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.165679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbfs\" (UniqueName: \"kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.165903 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.165742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.165903 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.165778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.166473 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.166450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.168290 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.168269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.175281 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.175252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbfs\" (UniqueName: \"kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs\") pod \"error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") " pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.227862 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.227803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.366872 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.366844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:05:10.369602 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:05:10.369565    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a666dfa_d384_4978_9207_ab37bf0c89a9.slice/crio-c6a6e2cdbc1031d7df50e14fb5e07623642a8b04256c33f4a6900a6f65f81c73 WatchSource:0}: Error finding container c6a6e2cdbc1031d7df50e14fb5e07623642a8b04256c33f4a6900a6f65f81c73: Status 404 returned error can't find the container with id c6a6e2cdbc1031d7df50e14fb5e07623642a8b04256c33f4a6900a6f65f81c73
Jul 28 16:05:10.560869 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.560830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f2ec7d-e239-4761-8142-35c1e124d730" path="/var/lib/kubelet/pods/10f2ec7d-e239-4761-8142-35c1e124d730/volumes"
Jul 28 16:05:10.653086 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.653002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:10.821601 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.821566    2577 generic.go:356] "Generic (PLEG): container finished" podID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerID="f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35" exitCode=2
Jul 28 16:05:10.821791 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.821644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerDied","Data":"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"}
Jul 28 16:05:10.823312 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.823287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerStarted","Data":"f4dca325580e116e1d4301c0f10ed0983013fba14e0f72accee38070fd557058"}
Jul 28 16:05:10.823452 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.823334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerStarted","Data":"3cc8e778e673b6b7c137c342f9771bbe25c0b062f1b0e9b052e67624c369951a"}
Jul 28 16:05:10.823452 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.823358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerStarted","Data":"c6a6e2cdbc1031d7df50e14fb5e07623642a8b04256c33f4a6900a6f65f81c73"}
Jul 28 16:05:10.823452 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.823437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:10.842608 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:10.842536    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podStartSLOduration=1.842514416 podStartE2EDuration="1.842514416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:10.841399509 +0000 UTC m=+1938.841639445" watchObservedRunningTime="2026-07-28 16:05:10.842514416 +0000 UTC m=+1938.842754343"
Jul 28 16:05:11.827501 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:11.827469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:11.828778 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:11.828746    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:12.831140 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:12.831087    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:13.010679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.010643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:13.143019 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.142984    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 16:05:13.291338 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.291298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6blg\" (UniqueName: \"kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg\") pod \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") "
Jul 28 16:05:13.291338 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.291347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") pod \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") "
Jul 28 16:05:13.291557 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.291398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config\") pod \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\" (UID: \"529d4acb-1ed1-46d2-a4cc-e89c1982008a\") "
Jul 28 16:05:13.291852 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.291821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "529d4acb-1ed1-46d2-a4cc-e89c1982008a" (UID: "529d4acb-1ed1-46d2-a4cc-e89c1982008a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:13.293483 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.293452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg" (OuterVolumeSpecName: "kube-api-access-z6blg") pod "529d4acb-1ed1-46d2-a4cc-e89c1982008a" (UID: "529d4acb-1ed1-46d2-a4cc-e89c1982008a"). InnerVolumeSpecName "kube-api-access-z6blg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:13.293589 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.293492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "529d4acb-1ed1-46d2-a4cc-e89c1982008a" (UID: "529d4acb-1ed1-46d2-a4cc-e89c1982008a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:13.397283 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.392347    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6blg\" (UniqueName: \"kubernetes.io/projected/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-api-access-z6blg\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:13.397283 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.392692    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/529d4acb-1ed1-46d2-a4cc-e89c1982008a-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:13.397283 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.392711    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/529d4acb-1ed1-46d2-a4cc-e89c1982008a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:13.835816 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.835773    2577 generic.go:356] "Generic (PLEG): container finished" podID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerID="d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3" exitCode=0
Jul 28 16:05:13.836266 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.835819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerDied","Data":"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"}
Jul 28 16:05:13.836266 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.835847    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"
Jul 28 16:05:13.836266 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.835881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295" event={"ID":"529d4acb-1ed1-46d2-a4cc-e89c1982008a","Type":"ContainerDied","Data":"4881466b27360c5c04f1d46d2e50a791f5bde0f46017f211c505197b8a2915c5"}
Jul 28 16:05:13.836266 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.835917    2577 scope.go:117] "RemoveContainer" containerID="f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"
Jul 28 16:05:13.845452 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.845433    2577 scope.go:117] "RemoveContainer" containerID="d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"
Jul 28 16:05:13.853083 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.853065    2577 scope.go:117] "RemoveContainer" containerID="f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"
Jul 28 16:05:13.853360 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:13.853339    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35\": container with ID starting with f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35 not found: ID does not exist" containerID="f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"
Jul 28 16:05:13.853447 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.853374    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35"} err="failed to get container status \"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35\": rpc error: code = NotFound desc = could not find container \"f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35\": container with ID starting with f68ead6d4bf1dd47d1e584bd81c14124d2c361c95f2f63855949a94444e93c35 not found: ID does not exist"
Jul 28 16:05:13.853447 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.853401    2577 scope.go:117] "RemoveContainer" containerID="d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"
Jul 28 16:05:13.853634 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:13.853616    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3\": container with ID starting with d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3 not found: ID does not exist" containerID="d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"
Jul 28 16:05:13.853673 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.853641    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3"} err="failed to get container status \"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3\": rpc error: code = NotFound desc = could not find container \"d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3\": container with ID starting with d20bb2f25fc0003078d56cbabd93bacaf205f248311dcb331ae830a00dd51ce3 not found: ID does not exist"
Jul 28 16:05:13.859552 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.859529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 16:05:13.863164 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:13.863140    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-50d42-predictor-8b88c967b-q9295"]
Jul 28 16:05:14.561555 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:14.561516    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" path="/var/lib/kubelet/pods/529d4acb-1ed1-46d2-a4cc-e89c1982008a/volumes"
Jul 28 16:05:17.712887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:17.712849    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 16:05:17.836199 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:17.836163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:17.836739 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:17.836709    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:18.010547 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:18.010509    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:23.009487 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:23.009445    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:23.009883 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:23.009547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 16:05:27.713238 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:27.713203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:05:27.837538 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:27.837498    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:28.009731 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:28.009696    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:33.010336 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:33.010298    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:37.837353 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:37.837308    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:38.009756 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:38.009718    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:39.763361 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.763335    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 16:05:39.787352 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.787320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") pod \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") "
Jul 28 16:05:39.787516 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.787362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle\") pod \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\" (UID: \"3dafbfb3-c9a9-4c29-93c3-aac3353e2452\") "
Jul 28 16:05:39.787798 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.787767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3dafbfb3-c9a9-4c29-93c3-aac3353e2452" (UID: "3dafbfb3-c9a9-4c29-93c3-aac3353e2452"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:39.789614 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.789577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dafbfb3-c9a9-4c29-93c3-aac3353e2452" (UID: "3dafbfb3-c9a9-4c29-93c3-aac3353e2452"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:39.790843 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.790820    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:05:39.791261 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791241    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container"
Jul 28 16:05:39.791306 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791266    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container"
Jul 28 16:05:39.791306 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kube-rbac-proxy"
Jul 28 16:05:39.791378 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791309    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kube-rbac-proxy"
Jul 28 16:05:39.791378 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42"
Jul 28 16:05:39.791378 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791332    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42"
Jul 28 16:05:39.791498 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kube-rbac-proxy"
Jul 28 16:05:39.791498 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791421    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="529d4acb-1ed1-46d2-a4cc-e89c1982008a" containerName="kserve-container"
Jul 28 16:05:39.791498 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.791431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerName="sequence-graph-50d42"
Jul 28 16:05:39.795660 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.794985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:39.795660 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.795579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:39.799536 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.799511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b87a7-serving-cert\""
Jul 28 16:05:39.799866 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.799847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b87a7-kube-rbac-proxy-sar-config\""
Jul 28 16:05:39.806421 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.806397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:05:39.888470 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.888387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:39.888470 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.888427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:39.888700 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.888484    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:39.888700 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.888495    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dafbfb3-c9a9-4c29-93c3-aac3353e2452-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:39.938507 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.938472    2577 generic.go:356] "Generic (PLEG): container finished" podID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" containerID="695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4" exitCode=0
Jul 28 16:05:39.938697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.938529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" event={"ID":"3dafbfb3-c9a9-4c29-93c3-aac3353e2452","Type":"ContainerDied","Data":"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"}
Jul 28 16:05:39.938697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.938557    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"
Jul 28 16:05:39.938697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.938579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx" event={"ID":"3dafbfb3-c9a9-4c29-93c3-aac3353e2452","Type":"ContainerDied","Data":"c674b83c053b8611d65ab84d23605d79605e79c91f1388616e5a74e6020a610e"}
Jul 28 16:05:39.938697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.938602    2577 scope.go:117] "RemoveContainer" containerID="695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"
Jul 28 16:05:39.948068 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.948045    2577 scope.go:117] "RemoveContainer" containerID="695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"
Jul 28 16:05:39.948346 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:39.948319    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4\": container with ID starting with 695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4 not found: ID does not exist" containerID="695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"
Jul 28 16:05:39.948416 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.948346    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4"} err="failed to get container status \"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4\": rpc error: code = NotFound desc = could not find container \"695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4\": container with ID starting with 695e7f33026d590b4cf61d915ca7d8ce63ca158da34e55c4b87ec39c6e1ab6a4 not found: ID does not exist"
Jul 28 16:05:39.962144 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.962115    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 16:05:39.966269 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.966242    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-50d42-799cd94864-7vggx"]
Jul 28 16:05:39.989257 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.989228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:39.989394 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.989259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:39.989394 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:39.989380    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-b87a7-serving-cert: secret "splitter-graph-b87a7-serving-cert" not found
Jul 28 16:05:39.989511 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:39.989456    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls podName:2b776be4-14a7-40af-82f1-9c44880461bf nodeName:}" failed. No retries permitted until 2026-07-28 16:05:40.489434316 +0000 UTC m=+1968.489674235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls") pod "splitter-graph-b87a7-758cb69b74-ndv9b" (UID: "2b776be4-14a7-40af-82f1-9c44880461bf") : secret "splitter-graph-b87a7-serving-cert" not found
Jul 28 16:05:39.989818 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:39.989800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:40.492998 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.492965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:40.495286 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.495259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") pod \"splitter-graph-b87a7-758cb69b74-ndv9b\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") " pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:40.560904 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.560870    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dafbfb3-c9a9-4c29-93c3-aac3353e2452" path="/var/lib/kubelet/pods/3dafbfb3-c9a9-4c29-93c3-aac3353e2452/volumes"
Jul 28 16:05:40.707918 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.707869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:40.846282 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.846248    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:05:40.849281 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:05:40.849252    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b776be4_14a7_40af_82f1_9c44880461bf.slice/crio-54ebcffd2007a5947a9d0f07843679d24bd95d4c444c322e6b0189b5db4342f2 WatchSource:0}: Error finding container 54ebcffd2007a5947a9d0f07843679d24bd95d4c444c322e6b0189b5db4342f2: Status 404 returned error can't find the container with id 54ebcffd2007a5947a9d0f07843679d24bd95d4c444c322e6b0189b5db4342f2
Jul 28 16:05:40.947754 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.947719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" event={"ID":"2b776be4-14a7-40af-82f1-9c44880461bf","Type":"ContainerStarted","Data":"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"}
Jul 28 16:05:40.947941 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.947783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" event={"ID":"2b776be4-14a7-40af-82f1-9c44880461bf","Type":"ContainerStarted","Data":"54ebcffd2007a5947a9d0f07843679d24bd95d4c444c322e6b0189b5db4342f2"}
Jul 28 16:05:40.947941 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.947811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:40.966764 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:40.966707    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podStartSLOduration=1.966687956 podStartE2EDuration="1.966687956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:40.965385247 +0000 UTC m=+1968.965625173" watchObservedRunningTime="2026-07-28 16:05:40.966687956 +0000 UTC m=+1968.966927974"
Jul 28 16:05:43.357972 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:43.357915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:46.957876 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:46.957800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:05:47.837645 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:47.837608    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 16:05:49.852637 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:49.852545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:05:49.853071 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:49.852729    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" containerID="cri-o://cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc" gracePeriod=30
Jul 28 16:05:49.873871 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:49.873820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.005871 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.005788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.034882 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.034850    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:05:50.035108 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.035062    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container" containerID="cri-o://434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1" gracePeriod=30
Jul 28 16:05:50.035257 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.035150    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kube-rbac-proxy" containerID="cri-o://a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559" gracePeriod=30
Jul 28 16:05:50.092845 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.092809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.158533 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.158495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.169993 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.169958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:05:50.173232 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.173211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.176055 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.176035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dd6ec-predictor-serving-cert\""
Jul 28 16:05:50.176263 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.176247    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dd6ec-kube-rbac-proxy-sar-config\""
Jul 28 16:05:50.183588 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.183565    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:05:50.264815 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.264773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.265005 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.264863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9pv\" (UniqueName: \"kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.265005 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.264931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.285359 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.285326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.366282 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.366244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9pv\" (UniqueName: \"kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.366464 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.366290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.366464 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.366324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.367036 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.367016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.368712 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.368688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.378078 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.378054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9pv\" (UniqueName: \"kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv\") pod \"error-404-isvc-dd6ec-predictor-d89d98c86-sswbf\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") " pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.386481 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.386447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:50.484382 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.484348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.614488 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.614455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:05:50.617761 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:05:50.617737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50225dc_6162_45f4_aa5e_d37676c111e8.slice/crio-513fde60a231e06651a43fb99342f455c555222277e32199a359da2aac9e1e34 WatchSource:0}: Error finding container 513fde60a231e06651a43fb99342f455c555222277e32199a359da2aac9e1e34: Status 404 returned error can't find the container with id 513fde60a231e06651a43fb99342f455c555222277e32199a359da2aac9e1e34
Jul 28 16:05:50.990646 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.990611    2577 generic.go:356] "Generic (PLEG): container finished" podID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerID="a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559" exitCode=2
Jul 28 16:05:50.991025 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.990675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerDied","Data":"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"}
Jul 28 16:05:50.992369 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.992350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerStarted","Data":"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"}
Jul 28 16:05:50.992494 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.992395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerStarted","Data":"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"}
Jul 28 16:05:50.992494 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.992415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerStarted","Data":"513fde60a231e06651a43fb99342f455c555222277e32199a359da2aac9e1e34"}
Jul 28 16:05:50.992494 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.992440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.992664 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.992500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:50.993898 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:50.993873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:05:51.012196 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:51.012151    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podStartSLOduration=1.012137876 podStartE2EDuration="1.012137876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:51.011719993 +0000 UTC m=+1979.011959920" watchObservedRunningTime="2026-07-28 16:05:51.012137876 +0000 UTC m=+1979.012377819"
Jul 28 16:05:51.955821 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:51.955781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:51.996953 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:51.996911    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:05:52.706732 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:52.706693    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 16:05:56.485856 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.485830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:05:56.512817 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.512732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh2xb\" (UniqueName: \"kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb\") pod \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") "
Jul 28 16:05:56.512817 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.512795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls\") pod \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") "
Jul 28 16:05:56.512817 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.512815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config\") pod \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\" (UID: \"95a1bb7a-5770-4953-abb6-d23aa5ecf273\") "
Jul 28 16:05:56.513268 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.513242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95a1bb7a-5770-4953-abb6-d23aa5ecf273" (UID: "95a1bb7a-5770-4953-abb6-d23aa5ecf273"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:56.514851 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.514830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95a1bb7a-5770-4953-abb6-d23aa5ecf273" (UID: "95a1bb7a-5770-4953-abb6-d23aa5ecf273"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:56.514851 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.514830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb" (OuterVolumeSpecName: "kube-api-access-wh2xb") pod "95a1bb7a-5770-4953-abb6-d23aa5ecf273" (UID: "95a1bb7a-5770-4953-abb6-d23aa5ecf273"). InnerVolumeSpecName "kube-api-access-wh2xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:56.613847 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.613813    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95a1bb7a-5770-4953-abb6-d23aa5ecf273-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:56.613847 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.613843    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:56.613847 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.613854    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2xb\" (UniqueName: \"kubernetes.io/projected/95a1bb7a-5770-4953-abb6-d23aa5ecf273-kube-api-access-wh2xb\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:05:56.955985 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:56.955892    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:05:57.001466 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.001439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:05:57.001900 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.001873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:05:57.016638 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.016607    2577 generic.go:356] "Generic (PLEG): container finished" podID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerID="434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1" exitCode=0
Jul 28 16:05:57.016765 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.016712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerDied","Data":"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"}
Jul 28 16:05:57.016811 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.016725    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"
Jul 28 16:05:57.016811 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.016791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp" event={"ID":"95a1bb7a-5770-4953-abb6-d23aa5ecf273","Type":"ContainerDied","Data":"9e09096aba3af754cb96a1b9e60749b2ab2f9ce38789e53e9bff1277503f6737"}
Jul 28 16:05:57.016873 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.016834    2577 scope.go:117] "RemoveContainer" containerID="a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"
Jul 28 16:05:57.026606 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.026581    2577 scope.go:117] "RemoveContainer" containerID="434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"
Jul 28 16:05:57.035445 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.035421    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:05:57.037420 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.037352    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b87a7-predictor-cc4c5464d-rvlzp"]
Jul 28 16:05:57.037519 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.037496    2577 scope.go:117] "RemoveContainer" containerID="a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"
Jul 28 16:05:57.038658 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:57.037940    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559\": container with ID starting with a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559 not found: ID does not exist" containerID="a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"
Jul 28 16:05:57.038658 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.037995    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559"} err="failed to get container status \"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559\": rpc error: code = NotFound desc = could not find container \"a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559\": container with ID starting with a40691587d217d48fe63feb175747ccada591d444caee27b138adc699794d559 not found: ID does not exist"
Jul 28 16:05:57.038658 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.038022    2577 scope.go:117] "RemoveContainer" containerID="434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"
Jul 28 16:05:57.038658 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:05:57.038450    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1\": container with ID starting with 434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1 not found: ID does not exist" containerID="434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"
Jul 28 16:05:57.038658 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.038477    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1"} err="failed to get container status \"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1\": rpc error: code = NotFound desc = could not find container \"434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1\": container with ID starting with 434c9d8b72d577fec83c0408baa483fe6de626167419c88051f776a8249892f1 not found: ID does not exist"
Jul 28 16:05:57.837261 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:57.837230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:05:58.562105 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:05:58.562057    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" path="/var/lib/kubelet/pods/95a1bb7a-5770-4953-abb6-d23aa5ecf273/volumes"
Jul 28 16:06:01.956261 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:01.956222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:06:01.956684 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:01.956325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:06:06.956046 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:06.956004    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:06:07.002573 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:07.002541    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:06:09.871211 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871465    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container"
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container"
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kube-rbac-proxy"
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871502    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kube-rbac-proxy"
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871547    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kserve-container"
Jul 28 16:06:09.871564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.871556    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a1bb7a-5770-4953-abb6-d23aa5ecf273" containerName="kube-rbac-proxy"
Jul 28 16:06:09.873360 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.873330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:06:09.875975 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.875957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:09.879984 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.879964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bd785-kube-rbac-proxy-sar-config\""
Jul 28 16:06:09.880124 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.879969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bd785-serving-cert\""
Jul 28 16:06:09.885388 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.885363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:06:09.909615 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.909579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:09.909777 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:09.909645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:10.010477 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:10.010441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:10.010648 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:10.010493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:10.010648 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:06:10.010609    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bd785-serving-cert: secret "switch-graph-bd785-serving-cert" not found
Jul 28 16:06:10.010722 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:06:10.010665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls podName:71c375ee-d13e-4b2c-9261-c0074d366f20 nodeName:}" failed. No retries permitted until 2026-07-28 16:06:10.510647325 +0000 UTC m=+1998.510887227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls") pod "switch-graph-bd785-64486d8c7-99n55" (UID: "71c375ee-d13e-4b2c-9261-c0074d366f20") : secret "switch-graph-bd785-serving-cert" not found
Jul 28 16:06:10.011123 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:10.011081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:10.514210 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:10.514176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:10.514397 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:06:10.514330    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bd785-serving-cert: secret "switch-graph-bd785-serving-cert" not found
Jul 28 16:06:10.514443 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:06:10.514406    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls podName:71c375ee-d13e-4b2c-9261-c0074d366f20 nodeName:}" failed. No retries permitted until 2026-07-28 16:06:11.514390453 +0000 UTC m=+1999.514630356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls") pod "switch-graph-bd785-64486d8c7-99n55" (UID: "71c375ee-d13e-4b2c-9261-c0074d366f20") : secret "switch-graph-bd785-serving-cert" not found
Jul 28 16:06:10.572107 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:10.572061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:06:11.522312 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:11.522273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:11.524697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:11.524675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"switch-graph-bd785-64486d8c7-99n55\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") " pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:11.686457 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:11.686415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:11.814041 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:11.814016    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:06:11.816567 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:06:11.816538    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c375ee_d13e_4b2c_9261_c0074d366f20.slice/crio-61c6a2dea45d62d5eb5e105bc270e3019622415ebbe4209616ce3129003880df WatchSource:0}: Error finding container 61c6a2dea45d62d5eb5e105bc270e3019622415ebbe4209616ce3129003880df: Status 404 returned error can't find the container with id 61c6a2dea45d62d5eb5e105bc270e3019622415ebbe4209616ce3129003880df
Jul 28 16:06:11.955859 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:11.955819    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:06:12.074705 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:12.074613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" event={"ID":"71c375ee-d13e-4b2c-9261-c0074d366f20","Type":"ContainerStarted","Data":"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"}
Jul 28 16:06:12.074705 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:12.074669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" event={"ID":"71c375ee-d13e-4b2c-9261-c0074d366f20","Type":"ContainerStarted","Data":"61c6a2dea45d62d5eb5e105bc270e3019622415ebbe4209616ce3129003880df"}
Jul 28 16:06:12.074918 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:12.074739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:12.092320 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:12.092274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podStartSLOduration=3.092258894 podStartE2EDuration="3.092258894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:06:12.091040097 +0000 UTC m=+2000.091280022" watchObservedRunningTime="2026-07-28 16:06:12.092258894 +0000 UTC m=+2000.092498819"
Jul 28 16:06:16.956570 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:16.956533    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:06:17.002326 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:17.002286    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:06:18.084605 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:18.084569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:06:19.993968 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:19.993943    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:06:20.094553 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.094511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") pod \"2b776be4-14a7-40af-82f1-9c44880461bf\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") "
Jul 28 16:06:20.094710 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.094560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle\") pod \"2b776be4-14a7-40af-82f1-9c44880461bf\" (UID: \"2b776be4-14a7-40af-82f1-9c44880461bf\") "
Jul 28 16:06:20.094952 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.094926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2b776be4-14a7-40af-82f1-9c44880461bf" (UID: "2b776be4-14a7-40af-82f1-9c44880461bf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:06:20.096653 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.096614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b776be4-14a7-40af-82f1-9c44880461bf" (UID: "2b776be4-14a7-40af-82f1-9c44880461bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:06:20.104311 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.104283    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b776be4-14a7-40af-82f1-9c44880461bf" containerID="cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc" exitCode=0
Jul 28 16:06:20.104430 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.104325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"
Jul 28 16:06:20.104430 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.104357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" event={"ID":"2b776be4-14a7-40af-82f1-9c44880461bf","Type":"ContainerDied","Data":"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"}
Jul 28 16:06:20.104430 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.104409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b" event={"ID":"2b776be4-14a7-40af-82f1-9c44880461bf","Type":"ContainerDied","Data":"54ebcffd2007a5947a9d0f07843679d24bd95d4c444c322e6b0189b5db4342f2"}
Jul 28 16:06:20.104555 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.104431    2577 scope.go:117] "RemoveContainer" containerID="cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"
Jul 28 16:06:20.112746 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.112728    2577 scope.go:117] "RemoveContainer" containerID="cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"
Jul 28 16:06:20.113013 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:06:20.112992    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc\": container with ID starting with cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc not found: ID does not exist" containerID="cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"
Jul 28 16:06:20.113079 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.113020    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc"} err="failed to get container status \"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc\": rpc error: code = NotFound desc = could not find container \"cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc\": container with ID starting with cf3e98429f75ef28bbf172846e7207aed770e9ff79878815eee3decbce61afdc not found: ID does not exist"
Jul 28 16:06:20.126210 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.126186    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:06:20.130294 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.130272    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b87a7-758cb69b74-ndv9b"]
Jul 28 16:06:20.195607 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.195566    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b776be4-14a7-40af-82f1-9c44880461bf-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:06:20.195607 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.195598    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b776be4-14a7-40af-82f1-9c44880461bf-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:06:20.561253 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:20.561220    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" path="/var/lib/kubelet/pods/2b776be4-14a7-40af-82f1-9c44880461bf/volumes"
Jul 28 16:06:27.002353 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:27.002310    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:06:37.002294 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:37.002259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:06:50.121948 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.121912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:06:50.122341 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.122226    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7"
Jul 28 16:06:50.122341 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.122238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7"
Jul 28 16:06:50.122341 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.122292    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b776be4-14a7-40af-82f1-9c44880461bf" containerName="splitter-graph-b87a7"
Jul 28 16:06:50.125128 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.125112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.127914 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.127890    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-dd6ec-serving-cert\""
Jul 28 16:06:50.128040 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.127918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-dd6ec-kube-rbac-proxy-sar-config\""
Jul 28 16:06:50.133448 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.133422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:06:50.135703 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.135684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:06:50.235402 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.235370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.235575 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.235419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.336237 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.336203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.336403 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.336243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.336831 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.336804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.338615 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.338590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls\") pod \"splitter-graph-dd6ec-8474d74dd-rx4fl\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") " pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.363728 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.363700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:06:50.436580 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.436495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:50.566542 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:50.566467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:06:50.569220 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:06:50.569191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb114e8_aac1_4ca8_abec_4af73f0b8b2e.slice/crio-8ad4f4a06ed3b01dcd67fd7b564bec4aa445e1fb2208fc637cf14a31c256bc45 WatchSource:0}: Error finding container 8ad4f4a06ed3b01dcd67fd7b564bec4aa445e1fb2208fc637cf14a31c256bc45: Status 404 returned error can't find the container with id 8ad4f4a06ed3b01dcd67fd7b564bec4aa445e1fb2208fc637cf14a31c256bc45
Jul 28 16:06:51.222658 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:51.222621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" event={"ID":"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e","Type":"ContainerStarted","Data":"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"}
Jul 28 16:06:51.223122 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:51.222675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" event={"ID":"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e","Type":"ContainerStarted","Data":"8ad4f4a06ed3b01dcd67fd7b564bec4aa445e1fb2208fc637cf14a31c256bc45"}
Jul 28 16:06:51.223122 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:51.222728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:06:51.242076 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:51.242027    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podStartSLOduration=1.242012621 podStartE2EDuration="1.242012621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:06:51.241007751 +0000 UTC m=+2039.241247676" watchObservedRunningTime="2026-07-28 16:06:51.242012621 +0000 UTC m=+2039.242252592"
Jul 28 16:06:57.231848 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:06:57.231817    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:07:17.921201 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:07:17.921122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:17.982484 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:07:17.982451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:52.492650 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:07:52.492622    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:07:52.495399 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:07:52.495379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:08:59.437156 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:08:59.437123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:17.962578 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:12:17.962532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:52.514026 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:12:52.513993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:12:52.517200 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:12:52.517169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:15:04.675743 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.675697    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:15:04.678125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.675962    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" containerID="cri-o://1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204" gracePeriod=30
Jul 28 16:15:04.754506 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.754473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:04.882427 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.882385    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:15:04.882732 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.882664    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" containerID="cri-o://f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f" gracePeriod=30
Jul 28 16:15:04.882876 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:04.882687    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kube-rbac-proxy" containerID="cri-o://582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655" gracePeriod=30
Jul 28 16:15:05.033106 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:05.033069    2577 generic.go:356] "Generic (PLEG): container finished" podID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerID="582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655" exitCode=2
Jul 28 16:15:05.033267 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:05.033140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerDied","Data":"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"}
Jul 28 16:15:06.598761 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:06.598726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:06.997345 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:06.997315    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 16:15:07.002664 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.002636    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 16:15:07.230382 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.230339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:07.725580 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.725558    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:15:07.874534 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.874450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls\") pod \"e50225dc-6162-45f4-aa5e-d37676c111e8\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") "
Jul 28 16:15:07.874534 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.874504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh9pv\" (UniqueName: \"kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv\") pod \"e50225dc-6162-45f4-aa5e-d37676c111e8\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") "
Jul 28 16:15:07.874767 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.874595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config\") pod \"e50225dc-6162-45f4-aa5e-d37676c111e8\" (UID: \"e50225dc-6162-45f4-aa5e-d37676c111e8\") "
Jul 28 16:15:07.874950 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.874928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e50225dc-6162-45f4-aa5e-d37676c111e8" (UID: "e50225dc-6162-45f4-aa5e-d37676c111e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:07.876508 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.876486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e50225dc-6162-45f4-aa5e-d37676c111e8" (UID: "e50225dc-6162-45f4-aa5e-d37676c111e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:07.876574 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.876530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv" (OuterVolumeSpecName: "kube-api-access-xh9pv") pod "e50225dc-6162-45f4-aa5e-d37676c111e8" (UID: "e50225dc-6162-45f4-aa5e-d37676c111e8"). InnerVolumeSpecName "kube-api-access-xh9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:15:07.975314 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.975278    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:15:07.975314 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.975308    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e50225dc-6162-45f4-aa5e-d37676c111e8-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:15:07.975314 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:07.975319    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9pv\" (UniqueName: \"kubernetes.io/projected/e50225dc-6162-45f4-aa5e-d37676c111e8-kube-api-access-xh9pv\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:15:08.051633 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.051593    2577 generic.go:356] "Generic (PLEG): container finished" podID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerID="f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f" exitCode=0
Jul 28 16:15:08.051840 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.051650    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"
Jul 28 16:15:08.051840 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.051653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerDied","Data":"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"}
Jul 28 16:15:08.051840 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.051808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf" event={"ID":"e50225dc-6162-45f4-aa5e-d37676c111e8","Type":"ContainerDied","Data":"513fde60a231e06651a43fb99342f455c555222277e32199a359da2aac9e1e34"}
Jul 28 16:15:08.051840 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.051837    2577 scope.go:117] "RemoveContainer" containerID="582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"
Jul 28 16:15:08.060744 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.060718    2577 scope.go:117] "RemoveContainer" containerID="f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"
Jul 28 16:15:08.067788 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.067767    2577 scope.go:117] "RemoveContainer" containerID="582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"
Jul 28 16:15:08.068059 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:15:08.068040    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655\": container with ID starting with 582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655 not found: ID does not exist" containerID="582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"
Jul 28 16:15:08.068199 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.068069    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655"} err="failed to get container status \"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655\": rpc error: code = NotFound desc = could not find container \"582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655\": container with ID starting with 582fadc44d437d86e7eaae9cd4b9f0c79cc66ae2d6cb78c037904a9c07810655 not found: ID does not exist"
Jul 28 16:15:08.068199 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.068087    2577 scope.go:117] "RemoveContainer" containerID="f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"
Jul 28 16:15:08.068363 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:15:08.068345    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f\": container with ID starting with f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f not found: ID does not exist" containerID="f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"
Jul 28 16:15:08.068403 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.068369    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f"} err="failed to get container status \"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f\": rpc error: code = NotFound desc = could not find container \"f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f\": container with ID starting with f25eedbee7e465355f8dfb9f1216a7815e05bb50033a1784d92a2d11b0b57f7f not found: ID does not exist"
Jul 28 16:15:08.075433 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.075411    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:15:08.081251 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.081229    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd6ec-predictor-d89d98c86-sswbf"]
Jul 28 16:15:08.561291 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:08.561252    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" path="/var/lib/kubelet/pods/e50225dc-6162-45f4-aa5e-d37676c111e8/volumes"
Jul 28 16:15:12.230512 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:12.230472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:17.230155 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:17.230114    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:17.230540 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:17.230219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:15:22.230707 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:22.230665    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:27.230803 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:27.230757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:32.230307 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:32.230262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:15:34.810981 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.810957    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:15:34.876851 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.876817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls\") pod \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") "
Jul 28 16:15:34.877016 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.876891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle\") pod \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\" (UID: \"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e\") "
Jul 28 16:15:34.877244 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.877220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" (UID: "ddb114e8-aac1-4ca8-abec-4af73f0b8b2e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:34.878897 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.878878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" (UID: "ddb114e8-aac1-4ca8-abec-4af73f0b8b2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:34.977549 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.977522    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:15:34.977549 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:34.977548    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.150506 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.150466    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerID="1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204" exitCode=0
Jul 28 16:15:35.150665 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.150518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"
Jul 28 16:15:35.150665 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.150547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" event={"ID":"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e","Type":"ContainerDied","Data":"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"}
Jul 28 16:15:35.150665 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.150604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl" event={"ID":"ddb114e8-aac1-4ca8-abec-4af73f0b8b2e","Type":"ContainerDied","Data":"8ad4f4a06ed3b01dcd67fd7b564bec4aa445e1fb2208fc637cf14a31c256bc45"}
Jul 28 16:15:35.150665 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.150628    2577 scope.go:117] "RemoveContainer" containerID="1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"
Jul 28 16:15:35.159081 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.159065    2577 scope.go:117] "RemoveContainer" containerID="1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"
Jul 28 16:15:35.159376 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:15:35.159351    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204\": container with ID starting with 1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204 not found: ID does not exist" containerID="1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"
Jul 28 16:15:35.159463 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.159384    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204"} err="failed to get container status \"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204\": rpc error: code = NotFound desc = could not find container \"1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204\": container with ID starting with 1f16e0ff463000ecc9244e858d84a3d78e72ac6a624158423e6125720d758204 not found: ID does not exist"
Jul 28 16:15:35.173345 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.173324    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:15:35.175217 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:35.175197    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-dd6ec-8474d74dd-rx4fl"]
Jul 28 16:15:36.561376 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:15:36.561343    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" path="/var/lib/kubelet/pods/ddb114e8-aac1-4ca8-abec-4af73f0b8b2e/volumes"
Jul 28 16:17:52.534564 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:17:52.534478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:17:52.538046 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:17:52.538025    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:18:22.340989 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:18:22.340953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:18:36.285691 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:18:36.285651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:29.151414 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.151330    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:22:29.153756 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.151589    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" containerID="cri-o://42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822" gracePeriod=30
Jul 28 16:22:29.252722 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.252664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:29.497757 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.497713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:22:29.497995 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.497964    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container" containerID="cri-o://3cc8e778e673b6b7c137c342f9771bbe25c0b062f1b0e9b052e67624c369951a" gracePeriod=30
Jul 28 16:22:29.498075 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:29.498037    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kube-rbac-proxy" containerID="cri-o://f4dca325580e116e1d4301c0f10ed0983013fba14e0f72accee38070fd557058" gracePeriod=30
Jul 28 16:22:30.528916 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:30.528874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:30.607284 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:30.607249    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerID="f4dca325580e116e1d4301c0f10ed0983013fba14e0f72accee38070fd557058" exitCode=2
Jul 28 16:22:30.607453 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:30.607319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerDied","Data":"f4dca325580e116e1d4301c0f10ed0983013fba14e0f72accee38070fd557058"}
Jul 28 16:22:31.084441 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:31.084407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:31.177421 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:31.177381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:32.616473 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.616433    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerID="3cc8e778e673b6b7c137c342f9771bbe25c0b062f1b0e9b052e67624c369951a" exitCode=0
Jul 28 16:22:32.616847 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.616503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerDied","Data":"3cc8e778e673b6b7c137c342f9771bbe25c0b062f1b0e9b052e67624c369951a"}
Jul 28 16:22:32.647948 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.647921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:22:32.836793 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.836742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls\") pod \"8a666dfa-d384-4978-9207-ab37bf0c89a9\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") "
Jul 28 16:22:32.836992 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.836857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnbfs\" (UniqueName: \"kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs\") pod \"8a666dfa-d384-4978-9207-ab37bf0c89a9\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") "
Jul 28 16:22:32.836992 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.836898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config\") pod \"8a666dfa-d384-4978-9207-ab37bf0c89a9\" (UID: \"8a666dfa-d384-4978-9207-ab37bf0c89a9\") "
Jul 28 16:22:32.837294 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.837269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a666dfa-d384-4978-9207-ab37bf0c89a9" (UID: "8a666dfa-d384-4978-9207-ab37bf0c89a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:32.838855 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.838826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a666dfa-d384-4978-9207-ab37bf0c89a9" (UID: "8a666dfa-d384-4978-9207-ab37bf0c89a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:32.838984 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.838949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs" (OuterVolumeSpecName: "kube-api-access-wnbfs") pod "8a666dfa-d384-4978-9207-ab37bf0c89a9" (UID: "8a666dfa-d384-4978-9207-ab37bf0c89a9"). InnerVolumeSpecName "kube-api-access-wnbfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:32.937669 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.937615    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:22:32.937669 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.937663    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a666dfa-d384-4978-9207-ab37bf0c89a9-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:22:32.937669 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:32.937679    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbfs\" (UniqueName: \"kubernetes.io/projected/8a666dfa-d384-4978-9207-ab37bf0c89a9-kube-api-access-wnbfs\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:22:33.082755 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.082715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:33.622181 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.622139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6" event={"ID":"8a666dfa-d384-4978-9207-ab37bf0c89a9","Type":"ContainerDied","Data":"c6a6e2cdbc1031d7df50e14fb5e07623642a8b04256c33f4a6900a6f65f81c73"}
Jul 28 16:22:33.622685 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.622214    2577 scope.go:117] "RemoveContainer" containerID="f4dca325580e116e1d4301c0f10ed0983013fba14e0f72accee38070fd557058"
Jul 28 16:22:33.622685 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.622158    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"
Jul 28 16:22:33.630668 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.630643    2577 scope.go:117] "RemoveContainer" containerID="3cc8e778e673b6b7c137c342f9771bbe25c0b062f1b0e9b052e67624c369951a"
Jul 28 16:22:33.644947 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.644918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:22:33.651556 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:33.651524    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bd785-predictor-755d7cb5cd-cd5b6"]
Jul 28 16:22:34.561382 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:34.561341    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" path="/var/lib/kubelet/pods/8a666dfa-d384-4978-9207-ab37bf0c89a9/volumes"
Jul 28 16:22:38.082628 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:38.082590    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:43.082291 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:43.082251    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:43.082697 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:43.082378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:22:46.880549 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:46.880516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:47.786455 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:47.786423    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:48.082597 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:48.082505    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:48.658764 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:48.658735    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:49.502377 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:49.502348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:50.370513 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:50.370481    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:51.196952 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:51.196926    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:52.013744 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:52.013713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:52.555069 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:52.555039    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:22:52.558378 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:52.558350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcw4d_0de70cee-1194-44a8-b74a-420221243eb4/ovn-acl-logging/0.log"
Jul 28 16:22:52.860931 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:52.860822    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:53.082909 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:53.082873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:53.704302 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:53.704250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:54.530526 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:54.530489    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:55.400640 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:55.400609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:56.310677 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:56.310649    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:57.236259 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:57.236227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:58.082820 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:58.082779    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:22:58.175359 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:58.175327    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bd785-64486d8c7-99n55_71c375ee-d13e-4b2c-9261-c0074d366f20/switch-graph-bd785/0.log"
Jul 28 16:22:59.294465 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.294440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:22:59.332423 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.332394    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle\") pod \"71c375ee-d13e-4b2c-9261-c0074d366f20\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") "
Jul 28 16:22:59.332593 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.332442    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") pod \"71c375ee-d13e-4b2c-9261-c0074d366f20\" (UID: \"71c375ee-d13e-4b2c-9261-c0074d366f20\") "
Jul 28 16:22:59.332813 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.332794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "71c375ee-d13e-4b2c-9261-c0074d366f20" (UID: "71c375ee-d13e-4b2c-9261-c0074d366f20"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:59.334464 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.334441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71c375ee-d13e-4b2c-9261-c0074d366f20" (UID: "71c375ee-d13e-4b2c-9261-c0074d366f20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:59.433394 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.433302    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c375ee-d13e-4b2c-9261-c0074d366f20-openshift-service-ca-bundle\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:22:59.433394 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.433335    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c375ee-d13e-4b2c-9261-c0074d366f20-proxy-tls\") on node \"ip-10-0-134-118.ec2.internal\" DevicePath \"\""
Jul 28 16:22:59.711066 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.710965    2577 generic.go:356] "Generic (PLEG): container finished" podID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerID="42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822" exitCode=0
Jul 28 16:22:59.711066 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.711017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"
Jul 28 16:22:59.711066 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.711055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" event={"ID":"71c375ee-d13e-4b2c-9261-c0074d366f20","Type":"ContainerDied","Data":"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"}
Jul 28 16:22:59.711383 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.711130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55" event={"ID":"71c375ee-d13e-4b2c-9261-c0074d366f20","Type":"ContainerDied","Data":"61c6a2dea45d62d5eb5e105bc270e3019622415ebbe4209616ce3129003880df"}
Jul 28 16:22:59.711383 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.711157    2577 scope.go:117] "RemoveContainer" containerID="42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"
Jul 28 16:22:59.719316 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.719295    2577 scope.go:117] "RemoveContainer" containerID="42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"
Jul 28 16:22:59.719585 ip-10-0-134-118 kubenswrapper[2577]: E0728 16:22:59.719560    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822\": container with ID starting with 42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822 not found: ID does not exist" containerID="42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"
Jul 28 16:22:59.719635 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.719595    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822"} err="failed to get container status \"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822\": rpc error: code = NotFound desc = could not find container \"42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822\": container with ID starting with 42197bb31f5b1208a6ea920380ba077e149c0b925956594e8e0879dd64ffb822 not found: ID does not exist"
Jul 28 16:22:59.737787 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.737757    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:22:59.748203 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:22:59.748171    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bd785-64486d8c7-99n55"]
Jul 28 16:23:00.560923 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:00.560891    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" path="/var/lib/kubelet/pods/71c375ee-d13e-4b2c-9261-c0074d366f20/volumes"
Jul 28 16:23:02.325756 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.325685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:02.480605 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480573    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bns78/must-gather-lschq"]
Jul 28 16:23:02.480841 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480829    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kube-rbac-proxy"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kube-rbac-proxy"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480851    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480857    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480871    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480880    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kube-rbac-proxy"
Jul 28 16:23:02.480887 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480885    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kube-rbac-proxy"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480891    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480896    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480902    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480909    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480946    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kube-rbac-proxy"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480953    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb114e8-aac1-4ca8-abec-4af73f0b8b2e" containerName="splitter-graph-dd6ec"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480961    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kube-rbac-proxy"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480966    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50225dc-6162-45f4-aa5e-d37676c111e8" containerName="kserve-container"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480972    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c375ee-d13e-4b2c-9261-c0074d366f20" containerName="switch-graph-bd785"
Jul 28 16:23:02.481125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.480978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a666dfa-d384-4978-9207-ab37bf0c89a9" containerName="kserve-container"
Jul 28 16:23:02.485434 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.485417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.488779 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.488744    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bns78\"/\"default-dockercfg-rvrtl\""
Jul 28 16:23:02.488926 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.488745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bns78\"/\"openshift-service-ca.crt\""
Jul 28 16:23:02.489917 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.489903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bns78\"/\"kube-root-ca.crt\""
Jul 28 16:23:02.499085 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.499057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/must-gather-lschq"]
Jul 28 16:23:02.499679 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.499647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:02.560166 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.560137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbq66\" (UniqueName: \"kubernetes.io/projected/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-kube-api-access-vbq66\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.560349 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.560214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-must-gather-output\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.594830 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.594746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:02.661414 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.661367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbq66\" (UniqueName: \"kubernetes.io/projected/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-kube-api-access-vbq66\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.661632 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.661480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-must-gather-output\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.661881 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.661854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-must-gather-output\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.670582 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.670547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbq66\" (UniqueName: \"kubernetes.io/projected/c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7-kube-api-access-vbq66\") pod \"must-gather-lschq\" (UID: \"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7\") " pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.794533 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.794493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/must-gather-lschq"
Jul 28 16:23:02.926252 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.926228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/must-gather-lschq"]
Jul 28 16:23:02.928479 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:23:02.928452    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3cd4d4f_9e0b_40d2_8d0b_74ea7d457ce7.slice/crio-4a123ed90c806978d60985b8b055926af083d993efeb1e4c5bb763da865df5f7 WatchSource:0}: Error finding container 4a123ed90c806978d60985b8b055926af083d993efeb1e4c5bb763da865df5f7: Status 404 returned error can't find the container with id 4a123ed90c806978d60985b8b055926af083d993efeb1e4c5bb763da865df5f7
Jul 28 16:23:02.930247 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:02.930230    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:23:03.726487 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:03.726458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/must-gather-lschq" event={"ID":"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7","Type":"ContainerStarted","Data":"4a123ed90c806978d60985b8b055926af083d993efeb1e4c5bb763da865df5f7"}
Jul 28 16:23:04.731720 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:04.731681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/must-gather-lschq" event={"ID":"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7","Type":"ContainerStarted","Data":"8ecbd740e4b2983ba15d2034e921a6700bd8cf1ae7a758270162ebdfd3af7618"}
Jul 28 16:23:04.732125 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:04.731735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/must-gather-lschq" event={"ID":"c3cd4d4f-9e0b-40d2-8d0b-74ea7d457ce7","Type":"ContainerStarted","Data":"3d30b0a1682c6ae8ca732bd0edec1652a9012e56acccab6eccf7aa397f3cb1f3"}
Jul 28 16:23:04.749699 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:04.749632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bns78/must-gather-lschq" podStartSLOduration=2.032097138 podStartE2EDuration="2.749608909s" totalImagesPullingTime="717.511771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:04.74884987 +0000 UTC m=+3012.749089796" watchObservedRunningTime="2026-07-28 16:23:04.749608909 +0000 UTC m=+3012.749848836"
Jul 28 16:23:05.062232 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:05.062141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2b2bp_019a3992-b0cd-4983-b0b6-57510bb4458c/global-pull-secret-syncer/0.log"
Jul 28 16:23:05.203741 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:05.203693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log"
Jul 28 16:23:05.300315 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:05.300266    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-118.ec2.internal_389edd964b978460b3dd3a4bae70f4b1/haproxy/0.log"
Jul 28 16:23:08.445112 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:08.445058    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:08.543769 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:08.543624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2p9tk_8f84d86c-9a2d-432c-b154-72723a223f65/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:10.436799 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:10.436758    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-v5pwb_4f37889f-e173-4483-ad9a-38e455c2954d/monitoring-plugin/0.log"
Jul 28 16:23:10.685904 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:10.685808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrs8d_1028cb4b-8c23-4a1d-875d-871e134b7358/node-exporter/0.log"
Jul 28 16:23:10.731248 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:10.731219    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrs8d_1028cb4b-8c23-4a1d-875d-871e134b7358/kube-rbac-proxy/0.log"
Jul 28 16:23:10.786156 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:10.786086    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrs8d_1028cb4b-8c23-4a1d-875d-871e134b7358/init-textfile/0.log"
Jul 28 16:23:12.889450 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.889410    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"]
Jul 28 16:23:12.894714 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.894679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:12.925490 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.925452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"]
Jul 28 16:23:12.947815 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.947780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84stk\" (UniqueName: \"kubernetes.io/projected/8cfa2716-d46d-4221-9738-a1eda728577e-kube-api-access-84stk\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:12.947815 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.947818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-lib-modules\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:12.948069 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.947843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-sys\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:12.948069 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.947867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-proc\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:12.948069 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:12.947993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-podres\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.048987 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.048936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84stk\" (UniqueName: \"kubernetes.io/projected/8cfa2716-d46d-4221-9738-a1eda728577e-kube-api-access-84stk\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049206 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.048997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-lib-modules\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049206 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-sys\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049206 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-proc\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049206 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-podres\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049429 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-sys\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049429 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-podres\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049429 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-lib-modules\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.049429 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.049221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cfa2716-d46d-4221-9738-a1eda728577e-proc\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.060011 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.059979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84stk\" (UniqueName: \"kubernetes.io/projected/8cfa2716-d46d-4221-9738-a1eda728577e-kube-api-access-84stk\") pod \"perf-node-gather-daemonset-4695c\" (UID: \"8cfa2716-d46d-4221-9738-a1eda728577e\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.205659 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.205555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:13.379903 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.379870    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"]
Jul 28 16:23:13.383179 ip-10-0-134-118 kubenswrapper[2577]: W0728 16:23:13.383147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8cfa2716_d46d_4221_9738_a1eda728577e.slice/crio-aef2967c655f28313dac5ce0b19df7d0fa84579aa20c6c15702111b8a3db0255 WatchSource:0}: Error finding container aef2967c655f28313dac5ce0b19df7d0fa84579aa20c6c15702111b8a3db0255: Status 404 returned error can't find the container with id aef2967c655f28313dac5ce0b19df7d0fa84579aa20c6c15702111b8a3db0255
Jul 28 16:23:13.778630 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.778594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c" event={"ID":"8cfa2716-d46d-4221-9738-a1eda728577e","Type":"ContainerStarted","Data":"13d5111a4f673cd58ab981f60462ef421da272b30edbecbb64733b3e07ba8e0b"}
Jul 28 16:23:13.778815 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.778658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c" event={"ID":"8cfa2716-d46d-4221-9738-a1eda728577e","Type":"ContainerStarted","Data":"aef2967c655f28313dac5ce0b19df7d0fa84579aa20c6c15702111b8a3db0255"}
Jul 28 16:23:13.778815 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:13.778738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:14.785523 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:14.785496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cdd4cd54-vs4gm_80690136-e0fa-4675-a09a-b6b727afe242/console/0.log"
Jul 28 16:23:16.203334 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:16.203307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7mgz_02dda83b-0c24-4d24-9aa0-5d7da07ca16b/dns/0.log"
Jul 28 16:23:16.244760 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:16.244728    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7mgz_02dda83b-0c24-4d24-9aa0-5d7da07ca16b/kube-rbac-proxy/0.log"
Jul 28 16:23:16.385301 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:16.385270    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-df56s_cdbd0202-abf5-4432-8dc4-1197b601c3e6/dns-node-resolver/0.log"
Jul 28 16:23:16.978275 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:16.978225    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hgcb_a03cfd2f-05a8-4157-87c4-8dc38753fede/node-ca/0.log"
Jul 28 16:23:17.866152 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:17.866128    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599f5d8cdf-2lpp5_402e1330-0b95-401f-8dbf-17c90f19df84/router/0.log"
Jul 28 16:23:18.291900 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:18.291873    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-njtkj_bf9c2b6f-bd30-4fc7-b4cc-a94f4b543454/serve-healthcheck-canary/0.log"
Jul 28 16:23:18.792161 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:18.792125    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ppfv_b32fbe8d-6805-444c-a3e5-261b6f6e828f/kube-rbac-proxy/0.log"
Jul 28 16:23:18.817853 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:18.817828    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ppfv_b32fbe8d-6805-444c-a3e5-261b6f6e828f/exporter/0.log"
Jul 28 16:23:18.843546 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:18.843525    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ppfv_b32fbe8d-6805-444c-a3e5-261b6f6e828f/extractor/0.log"
Jul 28 16:23:19.792141 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:19.792117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c"
Jul 28 16:23:19.817283 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:19.817231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-4695c" podStartSLOduration=7.817215679 podStartE2EDuration="7.817215679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:13.812569101 +0000 UTC m=+3021.812809026" watchObservedRunningTime="2026-07-28 16:23:19.817215679 +0000 UTC m=+3027.817455604"
Jul 28 16:23:21.313576 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:21.313543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85d8b8ddfc-fcnvf_53d1f1f3-1f1d-4963-89f8-77b2b89ea812/manager/0.log"
Jul 28 16:23:21.354968 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:21.354940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-tvg6g_c53c25d0-9847-41eb-9318-da1e4addbbd6/manager/0.log"
Jul 28 16:23:21.552967 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:21.552930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-s6m9w_98133422-887a-4247-ad15-139acc514e74/manager/0.log"
Jul 28 16:23:28.123283 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.123248    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:23:28.153349 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.153307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/egress-router-binary-copy/0.log"
Jul 28 16:23:28.189581 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.189547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/cni-plugins/0.log"
Jul 28 16:23:28.220325 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.220295    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/bond-cni-plugin/0.log"
Jul 28 16:23:28.249638 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.249608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/routeoverride-cni/0.log"
Jul 28 16:23:28.284462 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.284427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/whereabouts-cni-bincopy/0.log"
Jul 28 16:23:28.318016 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.317979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fmx6g_eafbb7e1-e47c-429d-90df-3165977d0ae8/whereabouts-cni/0.log"
Jul 28 16:23:28.664636 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.664600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v2pc9_a3716281-5742-4fe4-8a3d-36ff4b977445/kube-multus/0.log"
Jul 28 16:23:28.830955 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.830924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tk25s_b9e7bcf7-53f3-4c4b-97dd-5764f946f446/network-metrics-daemon/0.log"
Jul 28 16:23:28.867611 ip-10-0-134-118 kubenswrapper[2577]: I0728 16:23:28.867579    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tk25s_b9e7bcf7-53f3-4c4b-97dd-5764f946f446/kube-rbac-proxy/0.log"