Jul 28 16:19:12.812403 ip-10-0-139-46 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 16:19:12.812417 ip-10-0-139-46 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 16:19:12.812426 ip-10-0-139-46 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 16:19:12.812718 ip-10-0-139-46 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 16:19:22.863460 ip-10-0-139-46 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 16:19:22.863482 ip-10-0-139-46 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 33694ef50d85490fa3645c9f90671483 --
Jul 28 16:21:23.538609 ip-10-0-139-46 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 16:21:23.915920 ip-10-0-139-46 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:21:23.915920 ip-10-0-139-46 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 16:21:23.915920 ip-10-0-139-46 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:21:23.915920 ip-10-0-139-46 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 16:21:23.915920 ip-10-0-139-46 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:21:23.916682 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.915985    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 16:21:23.921529 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921508    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.921529 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921528    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921532    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921535    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921538    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921542    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921545    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921547    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921550    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921553    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921555    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921558    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921561    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921564    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921566    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921569    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921573    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921575    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921578    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921580    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921588    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.921598 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921591    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921593    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921596    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921599    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921603    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921607    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921611    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921614    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921617    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921619    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921622    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921625    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921628    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921631    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921633    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921636    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921639    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921641    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921644    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.922075 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921647    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921649    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921652    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921654    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921657    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921660    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921662    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921665    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921667    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921671    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921673    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921676    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921678    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921681    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921683    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921686    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921689    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921692    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921696    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921700    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.922549 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921703    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921705    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921708    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921710    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921713    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921715    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921718    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921720    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921723    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921725    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921728    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921731    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921734    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921736    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921739    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921741    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921743    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921746    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921748    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921751    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.923065 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921753    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921755    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921758    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921760    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921763    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921765    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921769    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921774    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921777    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921780    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921782    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921785    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921787    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921789    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921792    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921794    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921797    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921799    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.923583 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.921802    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923831    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923839    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923843    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923846    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923849    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923853    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923856    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923858    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923861    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923865    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923867    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923870    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923873    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923875    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923878    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923880    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923883    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923886    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923888    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.924018 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923891    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923893    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923896    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923899    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923901    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923904    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923906    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923909    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923911    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923913    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923917    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923919    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923922    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923924    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923928    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923931    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923933    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923936    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923939    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.924541 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923942    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923945    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923947    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923951    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923953    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923956    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923958    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923961    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923963    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923966    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923968    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923970    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923973    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923975    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923978    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923980    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923983    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923985    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923988    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923992    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.924993 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923995    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.923998    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924001    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924003    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924006    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924008    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924011    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924017    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924019    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924022    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924024    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924027    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924030    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924033    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924036    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924038    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924041    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924043    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924046    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924048    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.925524 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924051    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924053    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924056    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924058    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924061    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924063    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924066    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924068    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924072    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924074    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924077    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924079    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924082    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924086    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924090    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924093    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924096    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924098    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924101    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.926006 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924106    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924109    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924528    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924534    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924537    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924539    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924542    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924545    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924548    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924551    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924554    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924557    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924559    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924562    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924564    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924567    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924569    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924572    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924575    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924577    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.926488 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924580    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924582    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924585    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924587    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924590    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924593    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924596    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924598    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924600    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924603    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924605    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924608    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924611    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924614    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924617    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924619    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924623    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924627    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924630    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.926964 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924633    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924636    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924638    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924641    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924643    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924646    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924649    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924651    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924654    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924656    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924659    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924661    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924663    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924666    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924668    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924671    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924673    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924676    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924679    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924681    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.927426 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924683    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924686    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924688    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924691    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924693    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924696    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924699    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924702    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924704    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924707    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924709    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924714    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924718    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924721    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924724    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924727    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924730    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924734    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924737    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.927903 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924739    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924742    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924745    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924748    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924751    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924753    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924756    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924758    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924761    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924763    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924767    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924769    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924772    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924775    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924777    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924780    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924782    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924785    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924787    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.928381 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924790    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924792    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924795    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.924797    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924884    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924895    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924904    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924910    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924918    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924922    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924926    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924931    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924934    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924937    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924940    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924943    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924946    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924949    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924952    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924955    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924958    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924961    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924972    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924975    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 16:21:23.928832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924978    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924981    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924984    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924988    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924991    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924995    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.924999    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925004    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925007    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925010    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925013    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925016    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925020    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925023    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925025    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925028    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925031    2555 flags.go:64] FLAG: --enable-server="true"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925040    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925044    2555 flags.go:64] FLAG: --event-burst="100"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925048    2555 flags.go:64] FLAG: --event-qps="50"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925051    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925054    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925056    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925060    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 16:21:23.929409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925063    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925066    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925069    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925072    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925074    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925078    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925080    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925083    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925086    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925089    2555 flags.go:64] FLAG: --feature-gates=""
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925092    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925095    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925099    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925102    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925105    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925110    2555 flags.go:64] FLAG: --help="false"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925113    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-139-46.ec2.internal"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925116    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925119    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925122    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925125    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925128    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925131    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 16:21:23.929985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925134    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925137    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925140    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925148    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925151    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925154    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925157    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925160    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925163    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925165    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925168    2555 flags.go:64] FLAG: --lock-file=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925171    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925173    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925176    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925182    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925184    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925187    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925190    2555 flags.go:64] FLAG: --logging-format="text"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925193    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925196    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925198    2555 flags.go:64] FLAG: --manifest-url=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925201    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925205    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925208    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925216    2555 flags.go:64] FLAG: --max-pods="110"
Jul 28 16:21:23.930598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925219    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925222    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925225    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925228    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925231    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925234    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925237    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925244    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925247    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925250    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925253    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925261    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925267    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925269    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925272    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925275    2555 flags.go:64] FLAG: --port="10250"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925278    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925281    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08dd2b33c70eded44"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925284    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925287    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925304    2555 flags.go:64] FLAG: --register-node="true"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925307    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925310    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925313    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 28 16:21:23.931206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925316    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925319    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925323    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925326    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925329    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925332    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925335    2555 flags.go:64] FLAG: --runonce="false"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925339    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925344    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925347    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925350    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925353    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925356    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925359    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925362    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925365    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925368    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925370    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925373    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925376    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925380    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925382    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925388    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925391    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925393    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 16:21:23.931902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925398    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925400    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925403    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925406    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925409    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925412    2555 flags.go:64] FLAG: --v="2"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925415    2555 flags.go:64] FLAG: --version="false"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925419    2555 flags.go:64] FLAG: --vmodule=""
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925423    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.925426    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925568    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925571    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925574    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925577    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925581    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925585    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925588    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925590    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925593    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925596    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925599    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925602    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.932533 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925604    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925607    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925610    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925612    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925615    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925617    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925625    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925628    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925630    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925633    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925635    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925638    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925640    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925643    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925645    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925648    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925650    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925653    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925655    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925658    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.933062 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925660    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925663    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925666    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925669    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925674    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925679    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925681    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925684    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925687    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925689    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925692    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925695    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925697    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925700    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925702    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925705    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925707    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925710    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925712    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925715    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.933693 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925718    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925720    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925723    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925725    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925728    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925731    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925733    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925737    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925741    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925744    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925746    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925748    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925751    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925753    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925756    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925758    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925762    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925766    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925769    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.934190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925771    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925774    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925776    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925779    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925781    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925785    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925788    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925791    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925794    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925797    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925800    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925803    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925805    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925809    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925812    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925814    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925817    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925820    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925823    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.934689 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925825    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925828    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925831    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925833    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925836    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925839    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925841    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925844    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.925846    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.926711    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.933406    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.933423    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.933438    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.933447    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 16:21:23.935143 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933504    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933510    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933513    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933517    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933519    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933522    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933525    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933528    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933530    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933533    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933536    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933538    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933541    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933544    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933547    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933549    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933552    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933554    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933557    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933560    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.935511 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933562    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933565    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933568    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933570    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933573    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933575    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933578    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933580    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933583    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933588    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933590    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933593    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933595    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933598    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933600    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933603    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933606    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933609    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933611    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.935998 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933614    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933616    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933619    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933621    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933624    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933627    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933629    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933632    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933634    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933637    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933640    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933642    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933645    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933647    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933650    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933652    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933655    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933657    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933660    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933662    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.936491 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933664    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933669    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933673    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933676    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933679    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933683    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933686    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933688    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933691    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933694    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933696    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933699    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933701    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933705    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933709    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933712    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933715    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933718    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933721    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.936977 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933725    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933729    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933732    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933736    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933739    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933742    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933744    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933747    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933750    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933752    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933755    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933757    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933760    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933763    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933765    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933768    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933771    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933773    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933776    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933778    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.937538 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933781    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.933786    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933912    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933917    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933920    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933923    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933926    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933930    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933933    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933935    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933939    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933943    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933946    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933949    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:21:23.938050 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933952    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933956    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933958    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933961    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933964    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933966    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933969    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933971    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933974    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933978    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933981    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933983    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933986    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933989    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933992    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933994    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.933997    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934000    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934002    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:21:23.938410 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934005    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934009    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934012    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934015    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934017    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934021    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934024    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934027    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934029    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934032    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934035    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934037    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934040    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934042    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934045    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934048    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934051    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934053    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934056    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:21:23.938855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934058    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934060    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934063    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934065    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934068    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934071    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934073    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934076    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934078    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934081    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934083    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934086    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934089    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934091    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934094    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934096    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934099    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934101    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934104    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934106    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:21:23.939323 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934108    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934111    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934113    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934116    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934118    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934120    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934123    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934125    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934128    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934131    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934133    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934136    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934138    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934141    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934143    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934146    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934148    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934151    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934154    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934159    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:21:23.939934 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934162    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934164    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934167    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934169    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934172    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934175    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934177    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934180    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:23.934182    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.934187    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.934321    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.936220    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.937394    2555 server.go:1019] "Starting client certificate rotation"
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.937504    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:21:23.940495 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.937543    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:21:23.941582 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.941522    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:21:23.943359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.943306    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:21:23.953000 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.952974    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 28 16:21:23.955234 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.955219    2555 log.go:25] "Validated CRI v1 image API"
Jul 28 16:21:23.956115 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.956097    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 16:21:23.961094 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.961069    2555 fs.go:135] Filesystem UUIDs: map[571696ce-04f0-44a9-9ca6-04e51210e85b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcacbe6e-94bb-45f3-a980-08facf6577bb:/dev/nvme0n1p3]
Jul 28 16:21:23.961165 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.961092    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 16:21:23.965695 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.965671    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:21:23.966862 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.966749    2555 manager.go:216] Machine: {Timestamp:2026-07-28 16:21:23.965452156 +0000 UTC m=+0.307791306 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3063634 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a58a225e06d997895f5137d7a831 SystemUUID:ec22a58a-225e-06d9-9789-5f5137d7a831 BootID:33694ef5-0d85-490f-a364-5c9f90671483 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c1:45:d5:3c:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c1:45:d5:3c:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:22:9c:87:38:a8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 16:21:23.966862 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.966860    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 16:21:23.966967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.966952    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 16:21:23.969679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.969654    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 16:21:23.969840 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.969684    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-46.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 16:21:23.970812 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.970800    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 16:21:23.970848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.970815    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 16:21:23.970848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.970827    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:21:23.971345 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.971334    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:21:23.972145 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.972134    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 16:21:23.973873 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.973861    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:21:23.974001 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.973991    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 16:21:23.974036 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974022    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 16:21:23.974036 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974032    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 16:21:23.974101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974048    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 16:21:23.974101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974060    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 28 16:21:23.974101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974069    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 16:21:23.974537 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974521    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:21:23.974611 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.974546    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:21:23.976738 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.976714    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 16:21:23.979429 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.979405    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 16:21:23.979514 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.979449    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 16:21:23.980916 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.980897    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 16:21:23.981773 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.981737    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 16:21:23.982012 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982000    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 16:21:23.982058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982017    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 16:21:23.982058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982033    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 16:21:23.982058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982042    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 16:21:23.982058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982054    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982061    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982072    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982080    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982091    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982099    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982133    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 16:21:23.982164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.982141    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 16:21:23.985411 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.985396    2555 server.go:1268] "Started kubelet"
Jul 28 16:21:23.985546 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.985494    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 16:21:23.985598 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.985573    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 16:21:23.986174 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.985471    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 16:21:23.986241 ip-10-0-139-46 systemd[1]: Started Kubernetes Kubelet.
Jul 28 16:21:23.987863 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.987213    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 16:21:23.987863 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.987439    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-46.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 16:21:23.987863 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:23.987436    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 16:21:23.987863 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:23.987531    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 16:21:23.988738 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.988722    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 16:21:23.991360 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:23.991340    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 16:21:23.993721 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.993703    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 16:21:23.993813 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.993722    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 16:21:23.993877 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:23.992860    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-46.ec2.internal.18c68095fd31de7e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-46.ec2.internal,UID:ip-10-0-139-46.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-46.ec2.internal,},FirstTimestamp:2026-07-28 16:21:23.985366654 +0000 UTC m=+0.327705809,LastTimestamp:2026-07-28 16:21:23.985366654 +0000 UTC m=+0.327705809,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-46.ec2.internal,}"
Jul 28 16:21:23.993937 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:23.993877    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:23.994439 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994395    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 16:21:23.994534 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994450    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 16:21:23.994534 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994467    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 16:21:23.994631 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994575    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 16:21:23.994631 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994584    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 16:21:23.994737 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994721    2555 factory.go:55] Registering systemd factory
Jul 28 16:21:23.994778 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.994739    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 28 16:21:23.995129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.995107    2555 factory.go:153] Registering CRI-O factory
Jul 28 16:21:23.995129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.995120    2555 factory.go:223] Registration of the crio container factory successfully
Jul 28 16:21:23.995204 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.995179    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 16:21:23.995238 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.995212    2555 factory.go:103] Registering Raw factory
Jul 28 16:21:23.995238 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.995222    2555 manager.go:1196] Started watching for new ooms in manager
Jul 28 16:21:23.996024 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:23.996011    2555 manager.go:349] Starting recovery of all containers
Jul 28 16:21:24.001877 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.001846    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 16:21:24.001877 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.001859    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 16:21:24.002407 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.002379    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 16:21:24.009445 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.009418    2555 manager.go:354] Recovery completed
Jul 28 16:21:24.010478 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.010458    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vcwpw"
Jul 28 16:21:24.010856 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.010824    2555 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 16:21:24.014039 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.014023    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.016080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016061    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.016203 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016089    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.016203 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016099    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.016203 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016186    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vcwpw"
Jul 28 16:21:24.016669 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016655    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 16:21:24.016708 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016670    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 16:21:24.016708 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016687    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:21:24.016767 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.016726    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.017476 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.017460    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.017558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.017487    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.017558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.017499    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.018217 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.018147    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-46.ec2.internal.18c68095ff067cb0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-46.ec2.internal,UID:ip-10-0-139-46.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-46.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-46.ec2.internal,},FirstTimestamp:2026-07-28 16:21:24.016078 +0000 UTC m=+0.358417151,LastTimestamp:2026-07-28 16:21:24.016078 +0000 UTC m=+0.358417151,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-46.ec2.internal,}"
Jul 28 16:21:24.019017 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.019001    2555 policy_none.go:49] "None policy: Start"
Jul 28 16:21:24.019102 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.019021    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 16:21:24.019102 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.019038    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 16:21:24.020325 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.020175    2555 policy_none.go:47] "Start"
Jul 28 16:21:24.058906 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.058886    2555 manager.go:337] "Starting Device Plugin manager"
Jul 28 16:21:24.059023 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.058920    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 16:21:24.059023 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.058932    2555 server.go:87] "Starting device plugin registration server"
Jul 28 16:21:24.059245 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.059231    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 16:21:24.059316 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.059252    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 16:21:24.059407 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.059393    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 16:21:24.059485 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.059470    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 16:21:24.059485 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.059482    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 16:21:24.059991 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.059966    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 16:21:24.060076 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.060029    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.159560 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.159521    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.160820 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.160589    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.160820 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.160619    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.160820 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.160628    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.160820 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.160659    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.166030 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.165977    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.166030 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.166006    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-46.ec2.internal\": node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.178756 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.178726    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.208370 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.208344    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 16:21:24.208492 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.208391    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 16:21:24.208492 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.208419    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 16:21:24.208492 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.208458    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 16:21:24.210525 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.210500    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:21:24.279101 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.279058    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.309541 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.309503    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"]
Jul 28 16:21:24.309615 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.309594    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.311185 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.311171    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.311240 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.311201    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.311240 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.311210    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.312384 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.312370    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.312582 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.312566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.312656 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.312607    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.313123 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313108    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.313202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313113    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.313202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313157    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.313202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313168    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.313202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313139    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.313355 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.313208    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.314388 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.314372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.314473 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.314397    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:21:24.315006 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.314990    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:21:24.315080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.315021    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:21:24.315080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.315034    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:21:24.344067 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.344045    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-46.ec2.internal\" not found" node="ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.348423 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.348407    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-46.ec2.internal\" not found" node="ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.379645 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.379623    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.396669 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.396641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.396769 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.396670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39f953389fbb7b355964e9b9fe5538ef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-46.ec2.internal\" (UID: \"39f953389fbb7b355964e9b9fe5538ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.396769 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.396687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.479894 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.479814    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.497190 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.497259 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.497259 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39f953389fbb7b355964e9b9fe5538ef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-46.ec2.internal\" (UID: \"39f953389fbb7b355964e9b9fe5538ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.497342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497253    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.497342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23660c78fea953db7eac9d97b433c18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal\" (UID: \"c23660c78fea953db7eac9d97b433c18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.497342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.497257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39f953389fbb7b355964e9b9fe5538ef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-46.ec2.internal\" (UID: \"39f953389fbb7b355964e9b9fe5538ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.580381 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.580340    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:24.645878 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.645839    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.651858 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:24.651833    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:24.680630 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:24.680596    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.390879 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.390845    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.491452 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.491415    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.546941 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.546907    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 16:21:26.547372 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.547088    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:21:26.591644 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.591606    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.604860 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.604837    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:21:26.630248 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.630160    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 16:16:25 +0000 UTC" deadline="2028-03-23 16:36:56.262561648 +0000 UTC"
Jul 28 16:21:26.630248 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.630197    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14496h15m29.632368288s"
Jul 28 16:21:26.692512 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.692485    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.703107 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.703077    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:21:26.792687 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.792640    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.806762 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:26.806729    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f953389fbb7b355964e9b9fe5538ef.slice/crio-e822348574d55504b1a9da210938c23a9e440974db98879d460c4ef64769ffb1 WatchSource:0}: Error finding container e822348574d55504b1a9da210938c23a9e440974db98879d460c4ef64769ffb1: Status 404 returned error can't find the container with id e822348574d55504b1a9da210938c23a9e440974db98879d460c4ef64769ffb1
Jul 28 16:21:26.807070 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:26.807045    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23660c78fea953db7eac9d97b433c18.slice/crio-e266fea097f1a634d8b18331aeedca43fb1f47c8f4d4ca4b0ccb7e27e36cf06e WatchSource:0}: Error finding container e266fea097f1a634d8b18331aeedca43fb1f47c8f4d4ca4b0ccb7e27e36cf06e: Status 404 returned error can't find the container with id e266fea097f1a634d8b18331aeedca43fb1f47c8f4d4ca4b0ccb7e27e36cf06e
Jul 28 16:21:26.811726 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.811703    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:21:26.821732 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.821679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal" event={"ID":"39f953389fbb7b355964e9b9fe5538ef","Type":"ContainerStarted","Data":"e822348574d55504b1a9da210938c23a9e440974db98879d460c4ef64769ffb1"}
Jul 28 16:21:26.822668 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:26.822646    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal" event={"ID":"c23660c78fea953db7eac9d97b433c18","Type":"ContainerStarted","Data":"e266fea097f1a634d8b18331aeedca43fb1f47c8f4d4ca4b0ccb7e27e36cf06e"}
Jul 28 16:21:26.893324 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.893234    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:26.993811 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:26.993758    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-46.ec2.internal\" not found"
Jul 28 16:21:27.064704 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.064671    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:21:27.104487 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.104449    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:27.119320 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.119283    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:21:27.120546 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.120524    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal"
Jul 28 16:21:27.129087 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.129060    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:21:27.585523 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.585486    2555 apiserver.go:52] "Watching apiserver"
Jul 28 16:21:27.593811 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.593778    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 16:21:27.595865 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.595835    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9tqnm","kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal","openshift-dns/node-resolver-7q76r","openshift-image-registry/node-ca-p5lv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal","openshift-multus/multus-additional-cni-plugins-n2xw8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f","openshift-cluster-node-tuning-operator/tuned-g824h","openshift-multus/multus-gfcgw","openshift-multus/network-metrics-daemon-ljfnz","openshift-network-diagnostics/network-check-target-tgggh","openshift-network-operator/iptables-alerter-4br26","openshift-ovn-kubernetes/ovnkube-node-s9vmh","kube-system/global-pull-secret-syncer-2nvj5"]
Jul 28 16:21:27.598507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.598486    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.600857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.600911    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.600972    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.600997    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.600861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jwpxp\""
Jul 28 16:21:27.601135 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.601083    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:27.603439 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.603415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.605202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.605169    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 16:21:27.605891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.605868    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.605988 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.605913    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8jrq5\""
Jul 28 16:21:27.606153 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.606136    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.606516 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.606501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.608867 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.608850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.609029 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.608983    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 16:21:27.609132 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.609032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zscdj\""
Jul 28 16:21:27.609132 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.609062    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:27.609132 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.608880    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.609349 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.608927    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.611368 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.611345    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.613444 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.613424    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.613576 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.613470    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bv6h7\""
Jul 28 16:21:27.613783 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.613764    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 16:21:27.613916 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.613853    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 16:21:27.614022 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.613876    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.614173 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.614250 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614225    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 16:21:27.614438 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-sys-fs\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.614527 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614457    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7cz\" (UniqueName: \"kubernetes.io/projected/b6e4341d-8f19-4490-998a-99396c205e30-kube-api-access-8t7cz\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.614527 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80d9dc6-f175-44da-be32-99667ffe2331-tmp-dir\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.614633 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.614633 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614609    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32d5b350-ecea-466c-a4a2-198e0e8ee180-serviceca\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.614728 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614644    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-registration-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.614728 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614671    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a80d9dc6-f175-44da-be32-99667ffe2331-hosts-file\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.614728 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.614702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-device-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.616938 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.616912    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 16:21:27.617403 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.617204    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c89mm\""
Jul 28 16:21:27.618087 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618042    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:27.618544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z7l\" (UniqueName: \"kubernetes.io/projected/a80d9dc6-f175-44da-be32-99667ffe2331-kube-api-access-98z7l\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.618544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618264    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d5b350-ecea-466c-a4a2-198e0e8ee180-host\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.618544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrcg\" (UniqueName: \"kubernetes.io/projected/32d5b350-ecea-466c-a4a2-198e0e8ee180-kube-api-access-drrcg\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.618544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618455    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-socket-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.618544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.618497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.619173 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.619141    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.620856 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.620796    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:21:27.622521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.622057    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 16:21:27.622521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.622164    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 16:21:27.622521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.622327    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qt455\""
Jul 28 16:21:27.622521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.622353    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 16:21:27.622750 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.622569    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.623542 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.623167    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.623542 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.623193    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 16:21:27.623542 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.623362    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.624673 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.624656    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.624993 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.624975    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rs8jt\""
Jul 28 16:21:27.625205 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.625176    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.625287 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.625260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.625337 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.625322    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:27.628384 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.628366    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.630510 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.630490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.631140 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.631122    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 16:21:27.631237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.631162    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 16:21:27.631237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.631169    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b5d9w\""
Jul 28 16:21:27.633339 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.633317    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 16:21:27.633433 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.633338    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 16:21:27.633433 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.633355    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:21:27.633433 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.633339    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7bx46\""
Jul 28 16:21:27.649661 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.649633    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zwx8g"
Jul 28 16:21:27.658169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.658152    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zwx8g"
Jul 28 16:21:27.705851 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.705813    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 16:21:27.719351 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysconfig\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0686c156-0f69-48e5-91e9-18b32c83249e-agent-certs\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-systemd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719386    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-netd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-env-overrides\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-iptables-alerter-script\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-kubelet-config\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.719507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-netns\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719520    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-ovn\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-script-lib\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrr7\" (UniqueName: \"kubernetes.io/projected/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-kube-api-access-zsrr7\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-bin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-systemd\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-tmp\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42rx\" (UniqueName: \"kubernetes.io/projected/39385f3c-509b-4451-8d3c-7cd58f3599bd-kube-api-access-v42rx\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-os-release\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.719864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsctz\" (UniqueName: \"kubernetes.io/projected/3c5d987e-a8d3-4f77-949f-650857f72a9b-kube-api-access-dsctz\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-socket-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-sys-fs\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719936    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-daemon-config\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-sys-fs\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.719987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-run\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-socket-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-k8s-cni-cncf-io\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-slash\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-kubelet\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-device-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720178    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-device-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-log-socket\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720206    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80d9dc6-f175-44da-be32-99667ffe2331-tmp-dir\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-var-lib-kubelet\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-host\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-node-log\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32d5b350-ecea-466c-a4a2-198e0e8ee180-serviceca\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-system-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mv4d\" (UniqueName: \"kubernetes.io/projected/608e9c4b-1e1e-4f48-9513-7d2770221dd6-kube-api-access-4mv4d\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-sys\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0686c156-0f69-48e5-91e9-18b32c83249e-konnectivity-ca\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-netns\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-config\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80d9dc6-f175-44da-be32-99667ffe2331-tmp-dir\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.720800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a80d9dc6-f175-44da-be32-99667ffe2331-hosts-file\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a80d9dc6-f175-44da-be32-99667ffe2331-hosts-file\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720683    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-conf-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-kubelet\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-var-lib-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720763    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32d5b350-ecea-466c-a4a2-198e0e8ee180-serviceca\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-etc-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-host-slash\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-system-cni-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-os-release\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.720965    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cnibin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-kubernetes\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721049    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-systemd-units\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-bin\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/096be67a-895c-4ebb-a921-30f37159c2dc-kube-api-access-z6pm8\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.721449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7cz\" (UniqueName: \"kubernetes.io/projected/b6e4341d-8f19-4490-998a-99396c205e30-kube-api-access-8t7cz\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721222    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721267    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-tuned\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-multus-certs\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-modprobe-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-registration-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721462    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-hostroot\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4341d-8f19-4490-998a-99396c205e30-registration-dir\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-conf\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-lib-modules\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721551    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-multus\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721618    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z7l\" (UniqueName: \"kubernetes.io/projected/a80d9dc6-f175-44da-be32-99667ffe2331-kube-api-access-98z7l\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.722156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d5b350-ecea-466c-a4a2-198e0e8ee180-host\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcg\" (UniqueName: \"kubernetes.io/projected/32d5b350-ecea-466c-a4a2-198e0e8ee180-kube-api-access-drrcg\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d5b350-ecea-466c-a4a2-198e0e8ee180-host\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-cnibin\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cni-binary-copy\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.721972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-socket-dir-parent\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.722013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnhv\" (UniqueName: \"kubernetes.io/projected/e2c19cdc-09b8-450f-b38d-e34b65687d9f-kube-api-access-dwnhv\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.722038    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-dbus\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.722643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.722072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-etc-kubernetes\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.727282 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.727260    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:27.727282 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.727286    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:27.727433 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.727299    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:27.727433 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.727400    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:28.227379102 +0000 UTC m=+2.960008773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:27.728508 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.728457    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 16:21:27.733342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.733318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrcg\" (UniqueName: \"kubernetes.io/projected/32d5b350-ecea-466c-a4a2-198e0e8ee180-kube-api-access-drrcg\") pod \"node-ca-p5lv9\" (UID: \"32d5b350-ecea-466c-a4a2-198e0e8ee180\") " pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.733445 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.733424    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7cz\" (UniqueName: \"kubernetes.io/projected/b6e4341d-8f19-4490-998a-99396c205e30-kube-api-access-8t7cz\") pod \"aws-ebs-csi-driver-node-95f4f\" (UID: \"b6e4341d-8f19-4490-998a-99396c205e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.733593 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.733566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z7l\" (UniqueName: \"kubernetes.io/projected/a80d9dc6-f175-44da-be32-99667ffe2331-kube-api-access-98z7l\") pod \"node-resolver-7q76r\" (UID: \"a80d9dc6-f175-44da-be32-99667ffe2331\") " pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.822462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysconfig\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.822462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0686c156-0f69-48e5-91e9-18b32c83249e-agent-certs\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822483    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-systemd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-netd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-env-overrides\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-iptables-alerter-script\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-systemd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-kubelet-config\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-netd\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-netns\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-kubelet-config\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-ovn\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-script-lib\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.822679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrr7\" (UniqueName: \"kubernetes.io/projected/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-kube-api-access-zsrr7\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-ovn\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822664    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-netns\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-bin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-systemd\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822778    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-tmp\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42rx\" (UniqueName: \"kubernetes.io/projected/39385f3c-509b-4451-8d3c-7cd58f3599bd-kube-api-access-v42rx\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-os-release\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctz\" (UniqueName: \"kubernetes.io/projected/3c5d987e-a8d3-4f77-949f-650857f72a9b-kube-api-access-dsctz\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-daemon-config\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822927    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-run\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-k8s-cni-cncf-io\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.822997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-bin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-slash\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-slash\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.823135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823109    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-kubelet\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-env-overrides\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-log-socket\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-iptables-alerter-script\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-var-lib-kubelet\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-host\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-systemd\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823264    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-node-log\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-script-lib\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-run-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-log-socket\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-os-release\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-var-lib-kubelet\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823443    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.824187 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-kubelet\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823482    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-k8s-cni-cncf-io\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-host\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-system-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-node-log\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mv4d\" (UniqueName: \"kubernetes.io/projected/608e9c4b-1e1e-4f48-9513-7d2770221dd6-kube-api-access-4mv4d\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-sys\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-system-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0686c156-0f69-48e5-91e9-18b32c83249e-konnectivity-ca\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-netns\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-config\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.823672    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-conf-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.823747    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:28.32371389 +0000 UTC m=+3.056343563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823761    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-conf-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825020 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-kubelet\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-var-lib-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-etc-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823867    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-run\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-host-slash\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.823907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-system-cni-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-etc-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-netns\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824057    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysconfig\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-os-release\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-sys\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824144    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cnibin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-kubernetes\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-cni-dir\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-kubelet\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-systemd-units\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-os-release\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.825853 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.824262    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-host-slash\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0686c156-0f69-48e5-91e9-18b32c83249e-konnectivity-ca\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-bin\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-system-cni-dir\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/096be67a-895c-4ebb-a921-30f37159c2dc-kube-api-access-z6pm8\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-kubernetes\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824361    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-var-lib-openvswitch\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cnibin\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-systemd-units\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:27.824419    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:28.324401654 +0000 UTC m=+3.057031327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovnkube-config\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-cni-bin\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824575    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-daemon-config\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-tuned\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824625    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.826540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-multus-certs\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824697    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-modprobe-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824750    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-hostroot\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-conf\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-lib-modules\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824879    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-multus\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824947    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-modprobe-d\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.824990    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-lib-modules\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-sysctl-conf\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-hostroot\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-var-lib-cni-multus\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c5d987e-a8d3-4f77-949f-650857f72a9b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.827100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-host-run-multus-certs\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-cnibin\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cni-binary-copy\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825278    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-socket-dir-parent\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825487    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39385f3c-509b-4451-8d3c-7cd58f3599bd-cnibin\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825523    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnhv\" (UniqueName: \"kubernetes.io/projected/e2c19cdc-09b8-450f-b38d-e34b65687d9f-kube-api-access-dwnhv\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-dbus\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-etc-kubernetes\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-etc-kubernetes\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-dbus\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/608e9c4b-1e1e-4f48-9513-7d2770221dd6-multus-socket-dir-parent\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.825912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0686c156-0f69-48e5-91e9-18b32c83249e-agent-certs\") pod \"konnectivity-agent-9tqnm\" (UID: \"0686c156-0f69-48e5-91e9-18b32c83249e\") " pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.826058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39385f3c-509b-4451-8d3c-7cd58f3599bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.826149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-tmp\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.826185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608e9c4b-1e1e-4f48-9513-7d2770221dd6-cni-binary-copy\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.826992    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c5d987e-a8d3-4f77-949f-650857f72a9b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.827667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.827107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2c19cdc-09b8-450f-b38d-e34b65687d9f-etc-tuned\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.832902 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.832881    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsctz\" (UniqueName: \"kubernetes.io/projected/3c5d987e-a8d3-4f77-949f-650857f72a9b-kube-api-access-dsctz\") pod \"ovnkube-node-s9vmh\" (UID: \"3c5d987e-a8d3-4f77-949f-650857f72a9b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.833980 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.833960    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42rx\" (UniqueName: \"kubernetes.io/projected/39385f3c-509b-4451-8d3c-7cd58f3599bd-kube-api-access-v42rx\") pod \"multus-additional-cni-plugins-n2xw8\" (UID: \"39385f3c-509b-4451-8d3c-7cd58f3599bd\") " pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.834110 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.834026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnhv\" (UniqueName: \"kubernetes.io/projected/e2c19cdc-09b8-450f-b38d-e34b65687d9f-kube-api-access-dwnhv\") pod \"tuned-g824h\" (UID: \"e2c19cdc-09b8-450f-b38d-e34b65687d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.834273 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.834250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mv4d\" (UniqueName: \"kubernetes.io/projected/608e9c4b-1e1e-4f48-9513-7d2770221dd6-kube-api-access-4mv4d\") pod \"multus-gfcgw\" (UID: \"608e9c4b-1e1e-4f48-9513-7d2770221dd6\") " pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.834342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.834315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pm8\" (UniqueName: \"kubernetes.io/projected/096be67a-895c-4ebb-a921-30f37159c2dc-kube-api-access-z6pm8\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:27.835800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.835752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrr7\" (UniqueName: \"kubernetes.io/projected/5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7-kube-api-access-zsrr7\") pod \"iptables-alerter-4br26\" (UID: \"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7\") " pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:27.867885 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.867855    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:21:27.911164 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.911134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f"
Jul 28 16:21:27.921763 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.921740    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7q76r"
Jul 28 16:21:27.931557 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.931534    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p5lv9"
Jul 28 16:21:27.938207 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.938187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n2xw8"
Jul 28 16:21:27.945919 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.945897    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gfcgw"
Jul 28 16:21:27.954639 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.954621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:27.962266 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.962241    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g824h"
Jul 28 16:21:27.969892 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.969871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:27.983527 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:27.983505    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4br26"
Jul 28 16:21:28.228668 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.228585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:28.228837 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.228750    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:28.228837 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.228778    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:28.228837 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.228789    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:28.229020 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.228846    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:29.228827892 +0000 UTC m=+3.961457559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:28.329137 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.329108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:28.329321 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.329153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:28.329321 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.329269    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:28.329321 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.329272    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:28.329488 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.329331    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:29.329313349 +0000 UTC m=+4.061943005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:28.329488 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:28.329348    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:29.32933995 +0000 UTC m=+4.061969606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:28.661788 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.661748    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 16:16:27 +0000 UTC" deadline="2027-12-21 20:08:35.012388514 +0000 UTC"
Jul 28 16:21:28.661788 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.661781    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12267h47m6.350610368s"
Jul 28 16:21:28.690811 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:28.690726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39385f3c_509b_4451_8d3c_7cd58f3599bd.slice/crio-4e9e8ab9ad0b1fd4c08869f76791afdebd4b66270bb5c89a0c431cb1c070bec2 WatchSource:0}: Error finding container 4e9e8ab9ad0b1fd4c08869f76791afdebd4b66270bb5c89a0c431cb1c070bec2: Status 404 returned error can't find the container with id 4e9e8ab9ad0b1fd4c08869f76791afdebd4b66270bb5c89a0c431cb1c070bec2
Jul 28 16:21:28.695839 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:28.695811    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e9c4b_1e1e_4f48_9513_7d2770221dd6.slice/crio-0bf4f1bf47d5f667cb24701aae28b13bd181d24e477c2b750484da71ed130f0e WatchSource:0}: Error finding container 0bf4f1bf47d5f667cb24701aae28b13bd181d24e477c2b750484da71ed130f0e: Status 404 returned error can't find the container with id 0bf4f1bf47d5f667cb24701aae28b13bd181d24e477c2b750484da71ed130f0e
Jul 28 16:21:28.697081 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:28.696982    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80d9dc6_f175_44da_be32_99667ffe2331.slice/crio-7dacf3c93eccdbe4f28028ad24601ac5d5cdfcb671bbabc4a710a4265f8977ca WatchSource:0}: Error finding container 7dacf3c93eccdbe4f28028ad24601ac5d5cdfcb671bbabc4a710a4265f8977ca: Status 404 returned error can't find the container with id 7dacf3c93eccdbe4f28028ad24601ac5d5cdfcb671bbabc4a710a4265f8977ca
Jul 28 16:21:28.698573 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:28.698501    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c19cdc_09b8_450f_b38d_e34b65687d9f.slice/crio-1e269858b38a2252a60bddb589f84fb6e7d4c40d2a7626e54805d742f0443c6e WatchSource:0}: Error finding container 1e269858b38a2252a60bddb589f84fb6e7d4c40d2a7626e54805d742f0443c6e: Status 404 returned error can't find the container with id 1e269858b38a2252a60bddb589f84fb6e7d4c40d2a7626e54805d742f0443c6e
Jul 28 16:21:28.701474 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:21:28.701070    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5d987e_a8d3_4f77_949f_650857f72a9b.slice/crio-0eebd9bdbbdc117de8c65c719d9ce6e077a2b77320a10038dcae55ddc8d925c6 WatchSource:0}: Error finding container 0eebd9bdbbdc117de8c65c719d9ce6e077a2b77320a10038dcae55ddc8d925c6: Status 404 returned error can't find the container with id 0eebd9bdbbdc117de8c65c719d9ce6e077a2b77320a10038dcae55ddc8d925c6
Jul 28 16:21:28.827832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.827632    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f" event={"ID":"b6e4341d-8f19-4490-998a-99396c205e30","Type":"ContainerStarted","Data":"6f75002d22bfd6d3580cf206efcbb978bf610cdc10d0f40b0446f1dd7d275d4c"}
Jul 28 16:21:28.828751 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.828721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g824h" event={"ID":"e2c19cdc-09b8-450f-b38d-e34b65687d9f","Type":"ContainerStarted","Data":"1e269858b38a2252a60bddb589f84fb6e7d4c40d2a7626e54805d742f0443c6e"}
Jul 28 16:21:28.829817 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.829790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerStarted","Data":"4e9e8ab9ad0b1fd4c08869f76791afdebd4b66270bb5c89a0c431cb1c070bec2"}
Jul 28 16:21:28.831071 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.831040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p5lv9" event={"ID":"32d5b350-ecea-466c-a4a2-198e0e8ee180","Type":"ContainerStarted","Data":"6c191406ac8399399fe7d22584bb993c7f9771fa2527d6d8d863ed0437b9ed48"}
Jul 28 16:21:28.832215 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.832187    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4br26" event={"ID":"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7","Type":"ContainerStarted","Data":"9b693a309de9e481e8085d38344367f635291ec379948bf3556e855447c333d2"}
Jul 28 16:21:28.833420 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.833396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"0eebd9bdbbdc117de8c65c719d9ce6e077a2b77320a10038dcae55ddc8d925c6"}
Jul 28 16:21:28.834426 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.834401    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9tqnm" event={"ID":"0686c156-0f69-48e5-91e9-18b32c83249e","Type":"ContainerStarted","Data":"d7e14abf1a54f74458b65f828e43882f8e223b2d64f604b1b59ed10dd8f76ef2"}
Jul 28 16:21:28.835558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.835534    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7q76r" event={"ID":"a80d9dc6-f175-44da-be32-99667ffe2331","Type":"ContainerStarted","Data":"7dacf3c93eccdbe4f28028ad24601ac5d5cdfcb671bbabc4a710a4265f8977ca"}
Jul 28 16:21:28.836826 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.836774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gfcgw" event={"ID":"608e9c4b-1e1e-4f48-9513-7d2770221dd6","Type":"ContainerStarted","Data":"0bf4f1bf47d5f667cb24701aae28b13bd181d24e477c2b750484da71ed130f0e"}
Jul 28 16:21:28.839341 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.839300    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal" event={"ID":"39f953389fbb7b355964e9b9fe5538ef","Type":"ContainerStarted","Data":"1e5f9781be22e2c7d666bb6846817b22ca921e941b01d265961e728bd73911e0"}
Jul 28 16:21:28.853661 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:28.853576    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-46.ec2.internal" podStartSLOduration=1.853552896 podStartE2EDuration="1.853552896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:28.853331731 +0000 UTC m=+3.585961408" watchObservedRunningTime="2026-07-28 16:21:28.853552896 +0000 UTC m=+3.586182575"
Jul 28 16:21:29.234941 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.234858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:29.235121 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.235085    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:29.235121 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.235108    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:29.235121 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.235120    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:29.235352 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.235175    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:31.235157247 +0000 UTC m=+5.967786918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:29.336301 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.336268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:29.336460 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.336389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:29.336530 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.336512    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:29.336595 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.336581    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:31.33656311 +0000 UTC m=+6.069192781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:29.337041 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.336984    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:29.337125 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.337064    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:31.337039224 +0000 UTC m=+6.069668900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:29.662386 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.662319    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 16:16:27 +0000 UTC" deadline="2028-03-05 06:57:48.354240377 +0000 UTC"
Jul 28 16:21:29.662386 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.662384    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14054h36m18.691861146s"
Jul 28 16:21:29.821743 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.821148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:29.821743 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.821295    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:29.821743 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.821676    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:29.822067 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.821793    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:29.822067 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.821838    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:29.822067 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:29.821925    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:29.890487 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.890442    2555 generic.go:356] "Generic (PLEG): container finished" podID="c23660c78fea953db7eac9d97b433c18" containerID="ae3d35dc7d32a973ccc6b506d16a5742dab1a4acb1a64308d3ffc99f8e09dbbe" exitCode=0
Jul 28 16:21:29.891895 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:29.891591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal" event={"ID":"c23660c78fea953db7eac9d97b433c18","Type":"ContainerDied","Data":"ae3d35dc7d32a973ccc6b506d16a5742dab1a4acb1a64308d3ffc99f8e09dbbe"}
Jul 28 16:21:30.901118 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:30.901071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal" event={"ID":"c23660c78fea953db7eac9d97b433c18","Type":"ContainerStarted","Data":"c3aaf5285428cc6a3d29cb70624503c4214e3bafbd9ab4ce9078ba4948b0c4ab"}
Jul 28 16:21:31.254421 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.254276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:31.254631 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.254510    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:31.254631 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.254532    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:31.254631 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.254545    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:31.254631 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.254616    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:35.254596534 +0000 UTC m=+9.987226208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:31.355296 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.355225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:31.355296 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.355298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:31.355556 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.355408    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:31.355556 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.355456    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:35.355444751 +0000 UTC m=+10.088074406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:31.355873 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.355827    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:31.355994 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.355892    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:35.355866397 +0000 UTC m=+10.088496067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:31.821501 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.821470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:31.821656 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.821612    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:31.822038 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.821996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:31.822143 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.822105    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:31.822182 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:31.822165    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:31.822257 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:31.822236    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:33.820609 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:33.820570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:33.820946 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:33.820672    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:33.820946 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:33.820717    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:33.820946 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:33.820759    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:33.820946 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:33.820785    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:33.820946 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:33.820830    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:35.291037 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.290881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:35.291518 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.291093    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:35.291518 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.291112    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:35.291518 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.291122    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:35.291518 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.291179    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:43.291160764 +0000 UTC m=+18.023790422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:35.392073 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.392031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:35.392234 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.392100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:35.392234 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.392184    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:35.392234 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.392205    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:35.392396 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.392258    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:43.392240935 +0000 UTC m=+18.124870594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:35.392396 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.392275    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:43.392267861 +0000 UTC m=+18.124897517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:35.823080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.822269    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:35.823080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.822381    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:35.823080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:35.822438    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:35.823080 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.822582    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:35.823080 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.823023    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:35.823439 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:35.823189    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:37.818633 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:37.818594    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:37.818633 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:37.818628    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:37.819165 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:37.818645    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:37.819165 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:37.818740    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:37.819275 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:37.819168    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:37.819275 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:37.819231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:39.818848    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:39.818885    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:39.818920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:39.819033    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:39.819087    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:39.819354 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:39.819173    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:41.819266 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:41.819224    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:41.819266 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:41.819271    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:41.819746 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:41.819224    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:41.819746 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:41.819363    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:41.819746 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:41.819461    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:41.819746 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:41.819548    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:43.351119 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.351089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:43.351580 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.351216    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:43.351580 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.351232    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:43.351580 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.351241    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:43.351580 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.351295    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:59.351278669 +0000 UTC m=+34.083908344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:43.451626 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.451591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:43.451791 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.451639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:43.451791 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.451742    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:43.451791 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.451741    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:43.451959 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.451807    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:21:59.45178758 +0000 UTC m=+34.184417253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:43.451959 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.451825    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:59.451815511 +0000 UTC m=+34.184445170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:43.819242 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.819210    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:43.819242 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.819225    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:43.819242 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:43.819222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:43.819531 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.819335    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:43.819765 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.819720    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:43.819899 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:43.819814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:45.819776 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:45.819733    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:45.820261 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:45.819836    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:45.820261 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:45.819872    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:45.820261 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:45.819879    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:45.820261 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:45.819978    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:45.820261 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:45.820073    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:47.819464 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.819258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:47.820041 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.819272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:47.820041 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.819272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:47.820041 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:47.819615    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:47.820041 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:47.819538    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:47.820041 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:47.819721    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:47.938156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.938123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gfcgw" event={"ID":"608e9c4b-1e1e-4f48-9513-7d2770221dd6","Type":"ContainerStarted","Data":"5ae25f6f89ca18dcc1e5075d0a6997ee9c4dc9918860b684171fc72d619f869c"}
Jul 28 16:21:47.939866 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.939842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f" event={"ID":"b6e4341d-8f19-4490-998a-99396c205e30","Type":"ContainerStarted","Data":"203e0d8c7d231fd871593bbf48ddeb24b0ec5cceafc1c416edc84676827e2e71"}
Jul 28 16:21:47.941652 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.941626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g824h" event={"ID":"e2c19cdc-09b8-450f-b38d-e34b65687d9f","Type":"ContainerStarted","Data":"8017ac8fa279ac7b0099b776ec7dfef233e5eb18bdcdb4d0d377ec88e81292bb"}
Jul 28 16:21:47.943729 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.943705    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="44532f3c74beda1cc616ed5da77cc26d565a76c68998a2cbbe150ad735215938" exitCode=0
Jul 28 16:21:47.943955 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.943938    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"44532f3c74beda1cc616ed5da77cc26d565a76c68998a2cbbe150ad735215938"}
Jul 28 16:21:47.945789 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.945767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p5lv9" event={"ID":"32d5b350-ecea-466c-a4a2-198e0e8ee180","Type":"ContainerStarted","Data":"5a8a1480c6e0a01616b3d4d509524cc7210b3b5440a89544f3a350fa4f54b30f"}
Jul 28 16:21:47.951333 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.951255    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:21:47.952118 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952093    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c5d987e-a8d3-4f77-949f-650857f72a9b" containerID="0f6eef636ad7c8e9b808a46dd4594beb567101fca1b0e285760b81ebb754910b" exitCode=1
Jul 28 16:21:47.952233 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"48e5ed35fa7ff3724db1be1bcbcfdd7728ed37ca63c2a5797dcb2742ab5af1c7"}
Jul 28 16:21:47.952301 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952248    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"b9452bffb1a3c1285baeb77e6cbc5d99f47ded2b385d02d64faaef2ac4524f5b"}
Jul 28 16:21:47.952353 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"6fa864062d9a711051a262b4c38a5be80bc71c064536ae5bdd83323ef7c29541"}
Jul 28 16:21:47.952405 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"e60652cc5ee76eca35115449d7da13ac2431ec9088c52740c2aa8dffe235542f"}
Jul 28 16:21:47.952405 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerDied","Data":"0f6eef636ad7c8e9b808a46dd4594beb567101fca1b0e285760b81ebb754910b"}
Jul 28 16:21:47.952503 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.952427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"4633342e0a19611577533c40f5a7d880c7a37594e94205df24d0dcdac9ee690e"}
Jul 28 16:21:47.956536 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.956423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9tqnm" event={"ID":"0686c156-0f69-48e5-91e9-18b32c83249e","Type":"ContainerStarted","Data":"aa038d428b20c1fc618f5c1f7a536252cd4f327702fabf66c84bb7550fae8176"}
Jul 28 16:21:47.959697 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.957484    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-46.ec2.internal" podStartSLOduration=20.957468825 podStartE2EDuration="20.957468825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:30.920340713 +0000 UTC m=+5.652970393" watchObservedRunningTime="2026-07-28 16:21:47.957468825 +0000 UTC m=+22.690098505"
Jul 28 16:21:47.959697 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.957942    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gfcgw" podStartSLOduration=4.413362965 podStartE2EDuration="22.957933593s" totalImagesPullingTime="18.544570628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:47.953888965 +0000 UTC m=+22.686518648" watchObservedRunningTime="2026-07-28 16:21:47.957933593 +0000 UTC m=+22.690563272"
Jul 28 16:21:47.963891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.963609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7q76r" event={"ID":"a80d9dc6-f175-44da-be32-99667ffe2331","Type":"ContainerStarted","Data":"4feb063e5c42b02bee3b2f2524d16f439ab0b13b43fd20977bc029fd17d1042d"}
Jul 28 16:21:47.995789 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:47.995751    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g824h" podStartSLOduration=4.477222741 podStartE2EDuration="22.995738652s" totalImagesPullingTime="18.518515911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:47.995604302 +0000 UTC m=+22.728234005" watchObservedRunningTime="2026-07-28 16:21:47.995738652 +0000 UTC m=+22.728368330"
Jul 28 16:21:48.011810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.011763    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p5lv9" podStartSLOduration=4.520498807 podStartE2EDuration="23.011745648s" totalImagesPullingTime="18.491246841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:48.011645383 +0000 UTC m=+22.744275060" watchObservedRunningTime="2026-07-28 16:21:48.011745648 +0000 UTC m=+22.744375327"
Jul 28 16:21:48.026228 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.026180    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7q76r" podStartSLOduration=4.531376875 podStartE2EDuration="23.026164162s" totalImagesPullingTime="18.494787287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:48.025326417 +0000 UTC m=+22.757956095" watchObservedRunningTime="2026-07-28 16:21:48.026164162 +0000 UTC m=+22.758793841"
Jul 28 16:21:48.040149 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.039995    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9tqnm" podStartSLOduration=8.909222227 podStartE2EDuration="23.039979208s" totalImagesPullingTime="14.130756981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:48.039484137 +0000 UTC m=+22.772113818" watchObservedRunningTime="2026-07-28 16:21:48.039979208 +0000 UTC m=+22.772608888"
Jul 28 16:21:48.832969 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.832905    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 16:21:48.968044 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.967990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f" event={"ID":"b6e4341d-8f19-4490-998a-99396c205e30","Type":"ContainerStarted","Data":"d0213044814340f73ac013a15d1fb1689f6d1a479d215be1fbd392eaf110ab8f"}
Jul 28 16:21:48.969595 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.969556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4br26" event={"ID":"5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7","Type":"ContainerStarted","Data":"b3d161ef6261701676a110f06c64272352ddeb155f183e877e8c5033438848c0"}
Jul 28 16:21:48.984776 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:48.984731    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4br26" podStartSLOduration=5.5197341170000005 podStartE2EDuration="23.984718189s" totalImagesPullingTime="18.464984072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:48.984577631 +0000 UTC m=+23.717207310" watchObservedRunningTime="2026-07-28 16:21:48.984718189 +0000 UTC m=+23.717347867"
Jul 28 16:21:49.683448 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.683336    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T16:21:48.832920769Z","UUID":"bc011db6-ead5-4b5b-97cd-36641e5c4768","Handler":null,"Name":"","Endpoint":""}
Jul 28 16:21:49.684976 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.684955    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 16:21:49.685146 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.684997    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 16:21:49.819196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.819176    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:49.819196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.819190    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:49.819378 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.819176    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:49.819378 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:49.819308    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:49.819481 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:49.819376    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:49.819481 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:49.819436    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:49.974774 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.974674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f" event={"ID":"b6e4341d-8f19-4490-998a-99396c205e30","Type":"ContainerStarted","Data":"55501c9a7b14267988677bf5ec13bafb3d23deaa3bc9e7b8c62d30728ebb090e"}
Jul 28 16:21:49.991894 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:49.991840    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95f4f" podStartSLOduration=4.340650602 podStartE2EDuration="24.991821611s" totalImagesPullingTime="20.651171009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:49.991563404 +0000 UTC m=+24.724193081" watchObservedRunningTime="2026-07-28 16:21:49.991821611 +0000 UTC m=+24.724451290"
Jul 28 16:21:50.980315 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:50.980285    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:21:50.980741 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:50.980715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"eaca236cd1d60db8ecfdd7a8a0638c19a9274ab4d1731f0df6d3203811fb1446"}
Jul 28 16:21:51.818674 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:51.818642    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:51.818848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:51.818649    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:51.818848 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:51.818760    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:51.818848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:51.818650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:51.818848 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:51.818824    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:51.819093 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:51.818892    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:52.638676 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.638509    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:52.639447 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.639313    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:52.986929 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.986862    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="2e4b48a47d6e4db3c6c0c8915680658521eb94a325855fbf8e04b7aeb90178a1" exitCode=0
Jul 28 16:21:52.986929 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.986901    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"2e4b48a47d6e4db3c6c0c8915680658521eb94a325855fbf8e04b7aeb90178a1"}
Jul 28 16:21:52.994396 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.994375    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:21:52.994823 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.994795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"7660d45ce884f35bdeab749a48c9acac5dba776496876b67cd5e2d87d183cc57"}
Jul 28 16:21:52.994993 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.994977    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:52.995277 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.995248    2555 scope.go:117] "RemoveContainer" containerID="0f6eef636ad7c8e9b808a46dd4594beb567101fca1b0e285760b81ebb754910b"
Jul 28 16:21:52.995544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:52.995524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9tqnm"
Jul 28 16:21:53.818837 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:53.818687    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:53.819113 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:53.818698    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:53.819113 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:53.818922    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:53.819113 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:53.818698    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:53.819113 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:53.818980    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:53.819113 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:53.819067    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:53.998854 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:53.998824    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="8c2f1be15e0890b4f4feb42b858b1da99507bc0d4f848b297b831ab3da31e602" exitCode=0
Jul 28 16:21:53.998988 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:53.998915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"8c2f1be15e0890b4f4feb42b858b1da99507bc0d4f848b297b831ab3da31e602"}
Jul 28 16:21:54.002736 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.002723    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:21:54.003130 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.003107    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" event={"ID":"3c5d987e-a8d3-4f77-949f-650857f72a9b","Type":"ContainerStarted","Data":"bea2fda2cb42950258279c5a96c6d7b71df934adeddf4b539968b7cb71423bb1"}
Jul 28 16:21:54.003396 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.003331    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:54.003454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.003402    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:54.003454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.003433    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:54.018327 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.018308    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:54.018573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.018561    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:21:54.049243 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.049204    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh" podStartSLOduration=10.497529978 podStartE2EDuration="29.049187959s" totalImagesPullingTime="18.551657981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:54.047628825 +0000 UTC m=+28.780258502" watchObservedRunningTime="2026-07-28 16:21:54.049187959 +0000 UTC m=+28.781817638"
Jul 28 16:21:54.172396 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.172372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ljfnz"]
Jul 28 16:21:54.172539 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.172522    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:54.172662 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:54.172642    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:54.175536 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.175497    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgggh"]
Jul 28 16:21:54.175625 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.175597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:54.175706 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:54.175686    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:54.176129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.176113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2nvj5"]
Jul 28 16:21:54.176204 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:54.176193    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:54.176279 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:54.176260    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:55.008890 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:55.008827    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="aef1853bbd45accc6629cd55cfe29c10a277c8a7d2630b45f7adbbc7fc0d18b1" exitCode=0
Jul 28 16:21:55.009283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:55.008887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"aef1853bbd45accc6629cd55cfe29c10a277c8a7d2630b45f7adbbc7fc0d18b1"}
Jul 28 16:21:55.819642 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:55.819443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:55.819808 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:55.819518    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:55.819808 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:55.819740    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:55.819808 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:55.819558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:55.819971 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:55.819813    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:55.819971 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:55.819903    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:57.819202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:57.819175    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:57.819202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:57.819203    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:57.819648 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:57.819176    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:57.819648 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:57.819317    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:57.819648 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:57.819372    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:21:57.819648 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:57.819441    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:59.372402 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.372284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:59.372871 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.372447    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:21:59.372871 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.372471    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:21:59.372871 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.372486    2555 projected.go:196] Error preparing data for projected volume kube-api-access-dnbbr for pod openshift-network-diagnostics/network-check-target-tgggh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:59.372871 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.372540    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr podName:f516c902-f1cc-4df1-bf69-53b94ea9d874 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:31.372522086 +0000 UTC m=+66.105151762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnbbr" (UniqueName: "kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr") pod "network-check-target-tgggh" (UID: "f516c902-f1cc-4df1-bf69-53b94ea9d874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:21:59.473237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.473212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:59.473370 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.473247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:59.473370 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.473349    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:59.473370 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.473353    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:59.473526 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.473395    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs podName:096be67a-895c-4ebb-a921-30f37159c2dc nodeName:}" failed. No retries permitted until 2026-07-28 16:22:31.473382274 +0000 UTC m=+66.206011929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs") pod "network-metrics-daemon-ljfnz" (UID: "096be67a-895c-4ebb-a921-30f37159c2dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:21:59.473526 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.473407    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret podName:5bf00c4a-8cb3-48f4-a3d4-60e55b40d205 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:31.473401332 +0000 UTC m=+66.206030987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret") pod "global-pull-secret-syncer-2nvj5" (UID: "5bf00c4a-8cb3-48f4-a3d4-60e55b40d205") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:21:59.818693 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.818662    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:21:59.818856 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.818662    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:21:59.818856 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.818770    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ljfnz" podUID="096be67a-895c-4ebb-a921-30f37159c2dc"
Jul 28 16:21:59.818856 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:21:59.818662    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:21:59.818978 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.818863    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2nvj5" podUID="5bf00c4a-8cb3-48f4-a3d4-60e55b40d205"
Jul 28 16:21:59.818978 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:21:59.818915    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgggh" podUID="f516c902-f1cc-4df1-bf69-53b94ea9d874"
Jul 28 16:22:00.024132 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.024103    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="4755b53aeb180152404902cd0e5ae174cebeaa148577772c37ca5b026b68abc8" exitCode=0
Jul 28 16:22:00.024273 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.024139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"4755b53aeb180152404902cd0e5ae174cebeaa148577772c37ca5b026b68abc8"}
Jul 28 16:22:00.135517 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.135492    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-46.ec2.internal" event="NodeReady"
Jul 28 16:22:00.135645 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.135633    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 16:22:00.178541 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.178519    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w65fs"]
Jul 28 16:22:00.190415 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.190151    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b9s8s"]
Jul 28 16:22:00.190517 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.190439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.192692 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.192670    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 16:22:00.192789 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.192689    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 16:22:00.192789 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.192680    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hmx4q\""
Jul 28 16:22:00.210736 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.210718    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9s8s"]
Jul 28 16:22:00.210820 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.210745    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w65fs"]
Jul 28 16:22:00.210858 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.210823    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.212957 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.212940    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sllng\""
Jul 28 16:22:00.213064 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.212955    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 16:22:00.213064 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.212939    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 16:22:00.213064 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.212993    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 16:22:00.280305 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.280283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-kube-api-access-gvcdg\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.280515 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.280324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-config-volume\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.283030 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.280787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-tmp-dir\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.283030 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.280826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382028 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.381952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0834160-3b3d-49de-b396-5914cb537776-cert\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.382028 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.381996    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvr2h\" (UniqueName: \"kubernetes.io/projected/e0834160-3b3d-49de-b396-5914cb537776-kube-api-access-gvr2h\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-kube-api-access-gvcdg\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-config-volume\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-tmp-dir\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:00.382266    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:00.382317    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls podName:65c2aac8-86e5-45ac-aa53-7877cf0b3bb0 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:00.882300463 +0000 UTC m=+35.614930119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls") pod "dns-default-w65fs" (UID: "65c2aac8-86e5-45ac-aa53-7877cf0b3bb0") : secret "dns-default-metrics-tls" not found
Jul 28 16:22:00.382550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-tmp-dir\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.382769 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.382752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-config-volume\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.392141 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.392124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-kube-api-access-gvcdg\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.483079 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.483056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0834160-3b3d-49de-b396-5914cb537776-cert\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.483157 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.483088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvr2h\" (UniqueName: \"kubernetes.io/projected/e0834160-3b3d-49de-b396-5914cb537776-kube-api-access-gvr2h\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.485303 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.485287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0834160-3b3d-49de-b396-5914cb537776-cert\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.491172 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.491154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvr2h\" (UniqueName: \"kubernetes.io/projected/e0834160-3b3d-49de-b396-5914cb537776-kube-api-access-gvr2h\") pod \"ingress-canary-b9s8s\" (UID: \"e0834160-3b3d-49de-b396-5914cb537776\") " pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.516536 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.516514    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"]
Jul 28 16:22:00.541186 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.541162    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"]
Jul 28 16:22:00.541285 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.541247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"
Jul 28 16:22:00.543268 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.543250    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9s8s"
Jul 28 16:22:00.543723 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.543698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2kg8\""
Jul 28 16:22:00.543808 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.543749    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:22:00.543808 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.543761    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:22:00.684600 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.684430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhrn\" (UniqueName: \"kubernetes.io/projected/9f93020b-f7be-45c6-af6a-598acab297da-kube-api-access-wdhrn\") pod \"network-check-source-7d88cb55d8-457ph\" (UID: \"9f93020b-f7be-45c6-af6a-598acab297da\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"
Jul 28 16:22:00.706454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.706424    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9s8s"]
Jul 28 16:22:00.710188 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:00.710160    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0834160_3b3d_49de_b396_5914cb537776.slice/crio-a3f2ee04a79ed34f9eaf9f02410d5e26ce49b1e586ddcb799704bda19d94ca50 WatchSource:0}: Error finding container a3f2ee04a79ed34f9eaf9f02410d5e26ce49b1e586ddcb799704bda19d94ca50: Status 404 returned error can't find the container with id a3f2ee04a79ed34f9eaf9f02410d5e26ce49b1e586ddcb799704bda19d94ca50
Jul 28 16:22:00.785821 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.785791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhrn\" (UniqueName: \"kubernetes.io/projected/9f93020b-f7be-45c6-af6a-598acab297da-kube-api-access-wdhrn\") pod \"network-check-source-7d88cb55d8-457ph\" (UID: \"9f93020b-f7be-45c6-af6a-598acab297da\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"
Jul 28 16:22:00.794547 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.794529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhrn\" (UniqueName: \"kubernetes.io/projected/9f93020b-f7be-45c6-af6a-598acab297da-kube-api-access-wdhrn\") pod \"network-check-source-7d88cb55d8-457ph\" (UID: \"9f93020b-f7be-45c6-af6a-598acab297da\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"
Jul 28 16:22:00.849254 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.849233    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"
Jul 28 16:22:00.886588 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.886519    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:00.886684 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:00.886659    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:22:00.886765 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:00.886753    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls podName:65c2aac8-86e5-45ac-aa53-7877cf0b3bb0 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:01.886730166 +0000 UTC m=+36.619359910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls") pod "dns-default-w65fs" (UID: "65c2aac8-86e5-45ac-aa53-7877cf0b3bb0") : secret "dns-default-metrics-tls" not found
Jul 28 16:22:00.997589 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:00.997562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph"]
Jul 28 16:22:01.001029 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:01.000986    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f93020b_f7be_45c6_af6a_598acab297da.slice/crio-a926fcb22106a840021c7160c7d66b23605e7c7895e4d556266e1a8ce9fa88fc WatchSource:0}: Error finding container a926fcb22106a840021c7160c7d66b23605e7c7895e4d556266e1a8ce9fa88fc: Status 404 returned error can't find the container with id a926fcb22106a840021c7160c7d66b23605e7c7895e4d556266e1a8ce9fa88fc
Jul 28 16:22:01.027211 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.027181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph" event={"ID":"9f93020b-f7be-45c6-af6a-598acab297da","Type":"ContainerStarted","Data":"a926fcb22106a840021c7160c7d66b23605e7c7895e4d556266e1a8ce9fa88fc"}
Jul 28 16:22:01.030263 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.030239    2555 generic.go:356] "Generic (PLEG): container finished" podID="39385f3c-509b-4451-8d3c-7cd58f3599bd" containerID="6140072b8cd5207ed551815cc5ab4475be43166643cd441ed9241803885e5541" exitCode=0
Jul 28 16:22:01.030350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.030307    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerDied","Data":"6140072b8cd5207ed551815cc5ab4475be43166643cd441ed9241803885e5541"}
Jul 28 16:22:01.031367 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.031338    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9s8s" event={"ID":"e0834160-3b3d-49de-b396-5914cb537776","Type":"ContainerStarted","Data":"a3f2ee04a79ed34f9eaf9f02410d5e26ce49b1e586ddcb799704bda19d94ca50"}
Jul 28 16:22:01.818600 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.818556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:22:01.819299 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.818988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:22:01.819299 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.819154    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:22:01.821533 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.821349    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:22:01.822432 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.822404    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnm96\""
Jul 28 16:22:01.822647 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.822631    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:22:01.822718 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.822703    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr6mp\""
Jul 28 16:22:01.893841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:01.893811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:01.893976 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:01.893912    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:22:01.893976 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:01.893964    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls podName:65c2aac8-86e5-45ac-aa53-7877cf0b3bb0 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:03.893944716 +0000 UTC m=+38.626574386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls") pod "dns-default-w65fs" (UID: "65c2aac8-86e5-45ac-aa53-7877cf0b3bb0") : secret "dns-default-metrics-tls" not found
Jul 28 16:22:02.038454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.038298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" event={"ID":"39385f3c-509b-4451-8d3c-7cd58f3599bd","Type":"ContainerStarted","Data":"b4d01e494162f079b779628f0d197d5fb63586b358c42d5f2e7b53209e304101"}
Jul 28 16:22:02.060549 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.060497    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n2xw8" podStartSLOduration=8.576005216 podStartE2EDuration="37.060479515s" totalImagesPullingTime="28.484474299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:02.059215101 +0000 UTC m=+36.791844779" watchObservedRunningTime="2026-07-28 16:22:02.060479515 +0000 UTC m=+36.793109194"
Jul 28 16:22:02.538216 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.538159    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jxrnk"]
Jul 28 16:22:02.572359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.572330    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jxrnk"]
Jul 28 16:22:02.572521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.572460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.574897 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.574875    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lx2xj\""
Jul 28 16:22:02.575058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.574873    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 16:22:02.575806 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.575787    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 16:22:02.575894 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.575828    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 16:22:02.576026 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.575993    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 16:22:02.702437 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.702401    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbkv\" (UniqueName: \"kubernetes.io/projected/fd850d07-0f15-4189-a81e-46195dc699da-kube-api-access-wbbkv\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.702618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.702466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd850d07-0f15-4189-a81e-46195dc699da-data-volume\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.702618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.702603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.702757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.702681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd850d07-0f15-4189-a81e-46195dc699da-crio-socket\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.702757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.702720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804054 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.803953    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbkv\" (UniqueName: \"kubernetes.io/projected/fd850d07-0f15-4189-a81e-46195dc699da-kube-api-access-wbbkv\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804054 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd850d07-0f15-4189-a81e-46195dc699da-data-volume\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd850d07-0f15-4189-a81e-46195dc699da-crio-socket\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804272 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:02.804240    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:22:02.804446 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:02.804302    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:03.304283605 +0000 UTC m=+38.036913285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : configmap "kube-rbac-proxy" not found
Jul 28 16:22:02.804446 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:02.804405    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:02.804539 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:02.804463    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:03.304445777 +0000 UTC m=+38.037075446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:02.804686 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd850d07-0f15-4189-a81e-46195dc699da-crio-socket\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.804851 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.804830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd850d07-0f15-4189-a81e-46195dc699da-data-volume\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:02.818034 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:02.814467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbkv\" (UniqueName: \"kubernetes.io/projected/fd850d07-0f15-4189-a81e-46195dc699da-kube-api-access-wbbkv\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:03.307897 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.307856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:03.308394 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.307992    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:22:03.308394 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.308017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:03.308394 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.308080    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:04.308059279 +0000 UTC m=+39.040688937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : configmap "kube-rbac-proxy" not found
Jul 28 16:22:03.308394 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.308136    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:03.308394 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.308203    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:04.308188941 +0000 UTC m=+39.040818597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:03.806627 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.806595    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"]
Jul 28 16:22:03.826776 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.826747    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"
Jul 28 16:22:03.829191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.829160    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 16:22:03.829573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.829555    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 16:22:03.830401 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.830224    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k8spc\""
Jul 28 16:22:03.830724 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.830703    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"]
Jul 28 16:22:03.911179 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.911150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:03.911321 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:03.911203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmft\" (UniqueName: \"kubernetes.io/projected/eb780086-a4dc-4130-b43b-2b50e3b8ff01-kube-api-access-2xmft\") pod \"migrator-6865c9967-2h8xl\" (UID: \"eb780086-a4dc-4130-b43b-2b50e3b8ff01\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"
Jul 28 16:22:03.911321 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.911310    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:22:03.911424 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:03.911398    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls podName:65c2aac8-86e5-45ac-aa53-7877cf0b3bb0 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:07.911376238 +0000 UTC m=+42.644005894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls") pod "dns-default-w65fs" (UID: "65c2aac8-86e5-45ac-aa53-7877cf0b3bb0") : secret "dns-default-metrics-tls" not found
Jul 28 16:22:04.011708 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.011678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmft\" (UniqueName: \"kubernetes.io/projected/eb780086-a4dc-4130-b43b-2b50e3b8ff01-kube-api-access-2xmft\") pod \"migrator-6865c9967-2h8xl\" (UID: \"eb780086-a4dc-4130-b43b-2b50e3b8ff01\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"
Jul 28 16:22:04.021906 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.021884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmft\" (UniqueName: \"kubernetes.io/projected/eb780086-a4dc-4130-b43b-2b50e3b8ff01-kube-api-access-2xmft\") pod \"migrator-6865c9967-2h8xl\" (UID: \"eb780086-a4dc-4130-b43b-2b50e3b8ff01\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"
Jul 28 16:22:04.146486 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.146455    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"
Jul 28 16:22:04.312336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.312287    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7q76r_a80d9dc6-f175-44da-be32-99667ffe2331/dns-node-resolver/0.log"
Jul 28 16:22:04.313465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.313445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:04.313524 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.313513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:04.313582 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:04.313567    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:22:04.313616 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:04.313604    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:04.313644 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:04.313636    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:06.313620443 +0000 UTC m=+41.046250099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : configmap "kube-rbac-proxy" not found
Jul 28 16:22:04.313683 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:04.313654    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:06.313641663 +0000 UTC m=+41.046271321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:04.708960 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:04.708935    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl"]
Jul 28 16:22:04.712279 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:04.712238    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb780086_a4dc_4130_b43b_2b50e3b8ff01.slice/crio-75574c52f2e06b9befebd5d8194b928ba3eb23bdc20124e79d94026444323a73 WatchSource:0}: Error finding container 75574c52f2e06b9befebd5d8194b928ba3eb23bdc20124e79d94026444323a73: Status 404 returned error can't find the container with id 75574c52f2e06b9befebd5d8194b928ba3eb23bdc20124e79d94026444323a73
Jul 28 16:22:05.046680 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.046461    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9s8s" event={"ID":"e0834160-3b3d-49de-b396-5914cb537776","Type":"ContainerStarted","Data":"4d04909ec562623a58c45acb0e37f9453a544eacd49263e37ff7a99ebfb6a6a0"}
Jul 28 16:22:05.047827 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.047805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph" event={"ID":"9f93020b-f7be-45c6-af6a-598acab297da","Type":"ContainerStarted","Data":"4d07912b8fae90fb4a05cc997a05dcf4143736bcf9006a6550a24a1d08208a9c"}
Jul 28 16:22:05.049110 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.049092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl" event={"ID":"eb780086-a4dc-4130-b43b-2b50e3b8ff01","Type":"ContainerStarted","Data":"75574c52f2e06b9befebd5d8194b928ba3eb23bdc20124e79d94026444323a73"}
Jul 28 16:22:05.061855 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.061816    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b9s8s" podStartSLOduration=1.184756008 podStartE2EDuration="5.061795738s" totalImagesPullingTime="3.87703973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:05.060781999 +0000 UTC m=+39.793411694" watchObservedRunningTime="2026-07-28 16:22:05.061795738 +0000 UTC m=+39.794425415"
Jul 28 16:22:05.076380 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.076335    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-457ph" podStartSLOduration=1.481126623 podStartE2EDuration="5.076323027s" totalImagesPullingTime="3.595196404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:05.075529931 +0000 UTC m=+39.808159611" watchObservedRunningTime="2026-07-28 16:22:05.076323027 +0000 UTC m=+39.808952704"
Jul 28 16:22:05.712253 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:05.712228    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p5lv9_32d5b350-ecea-466c-a4a2-198e0e8ee180/node-ca/0.log"
Jul 28 16:22:06.112559 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:06.112531    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9s8s_e0834160-3b3d-49de-b396-5914cb537776/serve-healthcheck-canary/0.log"
Jul 28 16:22:06.324435 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:06.324403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:06.324588 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:06.324463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:06.324588 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:06.324527    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:06.324588 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:06.324543    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:22:06.324588 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:06.324585    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:10.32457019 +0000 UTC m=+45.057199845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:06.324750 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:06.324599    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:10.32459299 +0000 UTC m=+45.057222646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : configmap "kube-rbac-proxy" not found
Jul 28 16:22:07.056301 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:07.056252    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl" event={"ID":"eb780086-a4dc-4130-b43b-2b50e3b8ff01","Type":"ContainerStarted","Data":"258edcc38358aa07d02de3e4126ebd17586348c10f2e0e36655c16b1d5e25909"}
Jul 28 16:22:07.056727 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:07.056328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl" event={"ID":"eb780086-a4dc-4130-b43b-2b50e3b8ff01","Type":"ContainerStarted","Data":"ee73561998eef894ae1dd6ff3a38ee4f24aaef98046fedea107b73cff2931b7d"}
Jul 28 16:22:07.074657 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:07.074609    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2h8xl" podStartSLOduration=2.432855807 podStartE2EDuration="4.074595679s" totalImagesPullingTime="1.641739872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:07.073109124 +0000 UTC m=+41.805738801" watchObservedRunningTime="2026-07-28 16:22:07.074595679 +0000 UTC m=+41.807225356"
Jul 28 16:22:07.932708 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:07.932675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:07.932867 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:07.932828    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:22:07.932909 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:07.932903    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls podName:65c2aac8-86e5-45ac-aa53-7877cf0b3bb0 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:15.932885351 +0000 UTC m=+50.665515033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls") pod "dns-default-w65fs" (UID: "65c2aac8-86e5-45ac-aa53-7877cf0b3bb0") : secret "dns-default-metrics-tls" not found
Jul 28 16:22:10.346588 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:10.346549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:10.347040 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:10.346632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:10.347040 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:10.346683    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:22:10.347040 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:10.346737    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:10.347040 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:10.346758    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:18.346740471 +0000 UTC m=+53.079370131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : configmap "kube-rbac-proxy" not found
Jul 28 16:22:10.347040 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:10.346796    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:18.346783397 +0000 UTC m=+53.079413052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:15.987229 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:15.987197    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:15.990977 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:15.990949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c2aac8-86e5-45ac-aa53-7877cf0b3bb0-metrics-tls\") pod \"dns-default-w65fs\" (UID: \"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0\") " pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:16.139896 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:16.139868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:16.278138 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:16.277982    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w65fs"]
Jul 28 16:22:16.281746 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:16.281724    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c2aac8_86e5_45ac_aa53_7877cf0b3bb0.slice/crio-ded0fbee72af8a7bf889acc9db606a638a190ff50fe6ca92337a5239b8bc2ab6 WatchSource:0}: Error finding container ded0fbee72af8a7bf889acc9db606a638a190ff50fe6ca92337a5239b8bc2ab6: Status 404 returned error can't find the container with id ded0fbee72af8a7bf889acc9db606a638a190ff50fe6ca92337a5239b8bc2ab6
Jul 28 16:22:17.081090 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:17.081045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w65fs" event={"ID":"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0","Type":"ContainerStarted","Data":"ded0fbee72af8a7bf889acc9db606a638a190ff50fe6ca92337a5239b8bc2ab6"}
Jul 28 16:22:18.406637 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:18.406445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:18.407096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:18.406728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:18.407096 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:18.406858    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:18.407096 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:22:18.406912    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls podName:fd850d07-0f15-4189-a81e-46195dc699da nodeName:}" failed. No retries permitted until 2026-07-28 16:22:34.406898236 +0000 UTC m=+69.139527917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jxrnk" (UID: "fd850d07-0f15-4189-a81e-46195dc699da") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:22:18.407096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:18.406958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd850d07-0f15-4189-a81e-46195dc699da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:19.088898 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:19.088862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w65fs" event={"ID":"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0","Type":"ContainerStarted","Data":"99ad799e817762a113862a9c5cbc6037f9a927305495680ba4ce7a56123054a7"}
Jul 28 16:22:19.089096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:19.088910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w65fs" event={"ID":"65c2aac8-86e5-45ac-aa53-7877cf0b3bb0","Type":"ContainerStarted","Data":"78759594aab310234bdbc5aee7a3911b4d7b9390f85dd4d61e14674858d9a2a1"}
Jul 28 16:22:19.089096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:19.088994    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:19.105392 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:19.105352    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w65fs" podStartSLOduration=17.268052791 podStartE2EDuration="19.105338851s" totalImagesPullingTime="1.83728606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:19.103889635 +0000 UTC m=+53.836519313" watchObservedRunningTime="2026-07-28 16:22:19.105338851 +0000 UTC m=+53.837968528"
Jul 28 16:22:26.022646 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:26.022617    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9vmh"
Jul 28 16:22:29.094235 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.094205    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w65fs"
Jul 28 16:22:29.679833 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.679796    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-r849p"]
Jul 28 16:22:29.683018 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.682971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.683694 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.683665    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"]
Jul 28 16:22:29.685889 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.685863    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"]
Jul 28 16:22:29.686047 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.685992    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.687223 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.687196    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 16:22:29.688360 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.688333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.689344 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.689322    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 16:22:29.689478 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.689407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xvh4g\""
Jul 28 16:22:29.690856 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.690833    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 16:22:29.691251 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.691195    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 16:22:29.692100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.692069    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nmjcl\""
Jul 28 16:22:29.692339 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.692318    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 16:22:29.692894 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.692868    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 16:22:29.693102 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.693084    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 16:22:29.693263 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.693239    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 16:22:29.693348 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.693176    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 16:22:29.693348 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.693266    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 16:22:29.703083 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.703061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"]
Jul 28 16:22:29.703404 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.703388    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-r849p"]
Jul 28 16:22:29.704505 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.704484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"]
Jul 28 16:22:29.781810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlvn\" (UniqueName: \"kubernetes.io/projected/45cb79a9-7234-499b-93b2-26b628b5b9dd-kube-api-access-fvlvn\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.781913 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrvj\" (UniqueName: \"kubernetes.io/projected/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-kube-api-access-bsrvj\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.781913 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.782097 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45cb79a9-7234-499b-93b2-26b628b5b9dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.782097 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781965    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.782097 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.781984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.782097 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.782017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8c8383e-2bb0-435c-abf2-59869f817269-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.782314 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.782122    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.782314 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.782160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8c8383e-2bb0-435c-abf2-59869f817269-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.782314 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.782186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883108 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45cb79a9-7234-499b-93b2-26b628b5b9dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.883196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883129    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8c8383e-2bb0-435c-abf2-59869f817269-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.883196 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883390 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8c8383e-2bb0-435c-abf2-59869f817269-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.883390 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883390 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883259    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlvn\" (UniqueName: \"kubernetes.io/projected/45cb79a9-7234-499b-93b2-26b628b5b9dd-kube-api-access-fvlvn\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.883390 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrvj\" (UniqueName: \"kubernetes.io/projected/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-kube-api-access-bsrvj\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.883912 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.884082 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8c8383e-2bb0-435c-abf2-59869f817269-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.884210 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.883897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.887270 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.887373 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887302    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8c8383e-2bb0-435c-abf2-59869f817269-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-r849p\" (UID: \"e8c8383e-2bb0-435c-abf2-59869f817269\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:29.887465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-ca\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.887550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.887550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45cb79a9-7234-499b-93b2-26b628b5b9dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.887550 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.887484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.891353 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.891330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrvj\" (UniqueName: \"kubernetes.io/projected/2dc720a6-f197-4c3d-8ffc-ca0aedd447fd-kube-api-access-bsrvj\") pod \"cluster-proxy-proxy-agent-54ff858c4f-ldst8\" (UID: \"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:29.891427 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.891336    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlvn\" (UniqueName: \"kubernetes.io/projected/45cb79a9-7234-499b-93b2-26b628b5b9dd-kube-api-access-fvlvn\") pod \"managed-serviceaccount-addon-agent-5496bc9898-ts2qp\" (UID: \"45cb79a9-7234-499b-93b2-26b628b5b9dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:29.995091 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:29.995020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p"
Jul 28 16:22:30.010760 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:30.010737    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"
Jul 28 16:22:30.015112 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:30.015093    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"
Jul 28 16:22:30.164505 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:30.164479    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-r849p"]
Jul 28 16:22:30.165577 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:30.165551    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c8383e_2bb0_435c_abf2_59869f817269.slice/crio-dd7e63fd215322517920204e7005f01d631b8fe14a8d047df34ef777f7a546e8 WatchSource:0}: Error finding container dd7e63fd215322517920204e7005f01d631b8fe14a8d047df34ef777f7a546e8: Status 404 returned error can't find the container with id dd7e63fd215322517920204e7005f01d631b8fe14a8d047df34ef777f7a546e8
Jul 28 16:22:30.369854 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:30.369787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp"]
Jul 28 16:22:30.370757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:30.370736    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8"]
Jul 28 16:22:30.373181 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:30.373155    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc720a6_f197_4c3d_8ffc_ca0aedd447fd.slice/crio-8d418e1c4ea2d9de70fbf2676e83ae11bb6302a148a4e36b0a3322b9a2a4e968 WatchSource:0}: Error finding container 8d418e1c4ea2d9de70fbf2676e83ae11bb6302a148a4e36b0a3322b9a2a4e968: Status 404 returned error can't find the container with id 8d418e1c4ea2d9de70fbf2676e83ae11bb6302a148a4e36b0a3322b9a2a4e968
Jul 28 16:22:30.373836 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:30.373816    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45cb79a9_7234_499b_93b2_26b628b5b9dd.slice/crio-555a87dc1e095e79a50c6adfb40e5d687b98d87a8a0311a7d172f1180d7aedbc WatchSource:0}: Error finding container 555a87dc1e095e79a50c6adfb40e5d687b98d87a8a0311a7d172f1180d7aedbc: Status 404 returned error can't find the container with id 555a87dc1e095e79a50c6adfb40e5d687b98d87a8a0311a7d172f1180d7aedbc
Jul 28 16:22:31.128249 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.128183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8" event={"ID":"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd","Type":"ContainerStarted","Data":"8d418e1c4ea2d9de70fbf2676e83ae11bb6302a148a4e36b0a3322b9a2a4e968"}
Jul 28 16:22:31.129501 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.129447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp" event={"ID":"45cb79a9-7234-499b-93b2-26b628b5b9dd","Type":"ContainerStarted","Data":"555a87dc1e095e79a50c6adfb40e5d687b98d87a8a0311a7d172f1180d7aedbc"}
Jul 28 16:22:31.130949 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.130894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p" event={"ID":"e8c8383e-2bb0-435c-abf2-59869f817269","Type":"ContainerStarted","Data":"dd7e63fd215322517920204e7005f01d631b8fe14a8d047df34ef777f7a546e8"}
Jul 28 16:22:31.398896 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.398313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:22:31.407125 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.407094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/f516c902-f1cc-4df1-bf69-53b94ea9d874-kube-api-access-dnbbr\") pod \"network-check-target-tgggh\" (UID: \"f516c902-f1cc-4df1-bf69-53b94ea9d874\") " pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:22:31.499440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.499407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:22:31.499440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.499448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:22:31.502140 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.502113    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:22:31.502436 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.502369    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:22:31.517731 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.517705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf00c4a-8cb3-48f4-a3d4-60e55b40d205-original-pull-secret\") pod \"global-pull-secret-syncer-2nvj5\" (UID: \"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205\") " pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:22:31.519202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.519157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096be67a-895c-4ebb-a921-30f37159c2dc-metrics-certs\") pod \"network-metrics-daemon-ljfnz\" (UID: \"096be67a-895c-4ebb-a921-30f37159c2dc\") " pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:22:31.533142 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.533119    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2nvj5"
Jul 28 16:22:31.543669 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.543471    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr6mp\""
Jul 28 16:22:31.553034 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.549696    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnm96\""
Jul 28 16:22:31.553034 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.551581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ljfnz"
Jul 28 16:22:31.557912 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:31.557892    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:22:32.098718 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.098612    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2nvj5"]
Jul 28 16:22:32.109476 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.108914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgggh"]
Jul 28 16:22:32.131562 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.131466    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ljfnz"]
Jul 28 16:22:32.138550 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:32.138516    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096be67a_895c_4ebb_a921_30f37159c2dc.slice/crio-8ae2b57115161612f2bff9313abc853c3155c62dadefc32fa94074051064613d WatchSource:0}: Error finding container 8ae2b57115161612f2bff9313abc853c3155c62dadefc32fa94074051064613d: Status 404 returned error can't find the container with id 8ae2b57115161612f2bff9313abc853c3155c62dadefc32fa94074051064613d
Jul 28 16:22:32.141908 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.141193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p" event={"ID":"e8c8383e-2bb0-435c-abf2-59869f817269","Type":"ContainerStarted","Data":"fa18fb12ed8b6cbaccd22923c3735de53a3013af024971679290933db9432e69"}
Jul 28 16:22:32.147990 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.147963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2nvj5" event={"ID":"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205","Type":"ContainerStarted","Data":"bf86b934325259276a60a8803aa65f0d3b305d0a45c11d2cac92257e0b2c8cdb"}
Jul 28 16:22:32.152681 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.152624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgggh" event={"ID":"f516c902-f1cc-4df1-bf69-53b94ea9d874","Type":"ContainerStarted","Data":"f0c767cfe0621a6c4e4ae6524763dd7c5a0f6ca89c3cadf5c2c12c3807a9b774"}
Jul 28 16:22:32.159023 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:32.158914    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-r849p" podStartSLOduration=1.403492727 podStartE2EDuration="3.158894299s" totalImagesPullingTime="1.755401572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:32.15831516 +0000 UTC m=+66.890944857" watchObservedRunningTime="2026-07-28 16:22:32.158894299 +0000 UTC m=+66.891523977"
Jul 28 16:22:33.159794 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:33.159749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ljfnz" event={"ID":"096be67a-895c-4ebb-a921-30f37159c2dc","Type":"ContainerStarted","Data":"8ae2b57115161612f2bff9313abc853c3155c62dadefc32fa94074051064613d"}
Jul 28 16:22:33.162605 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:33.162345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgggh" event={"ID":"f516c902-f1cc-4df1-bf69-53b94ea9d874","Type":"ContainerStarted","Data":"f6d8b1ba0b85197dbab23f6271643c898ff4d1a184f6c149ceaddbb7ce92203c"}
Jul 28 16:22:33.162605 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:33.162464    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:22:33.179898 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:33.178952    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgggh" podStartSLOduration=68.178932777 podStartE2EDuration="1m8.178932777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:33.176790685 +0000 UTC m=+67.909420364" watchObservedRunningTime="2026-07-28 16:22:33.178932777 +0000 UTC m=+67.911562457"
Jul 28 16:22:34.429895 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:34.429366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:34.432419 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:34.432387    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd850d07-0f15-4189-a81e-46195dc699da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxrnk\" (UID: \"fd850d07-0f15-4189-a81e-46195dc699da\") " pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:34.686701 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:34.686616    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lx2xj\""
Jul 28 16:22:34.694678 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:34.694647    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jxrnk"
Jul 28 16:22:35.256454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:35.256431    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jxrnk"]
Jul 28 16:22:35.260043 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:35.259996    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd850d07_0f15_4189_a81e_46195dc699da.slice/crio-a548ac52de7ea37d0854db0d78dc3f0c2045d8838ffa4d88a78b91517c250b0c WatchSource:0}: Error finding container a548ac52de7ea37d0854db0d78dc3f0c2045d8838ffa4d88a78b91517c250b0c: Status 404 returned error can't find the container with id a548ac52de7ea37d0854db0d78dc3f0c2045d8838ffa4d88a78b91517c250b0c
Jul 28 16:22:36.176576 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.176535    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8" event={"ID":"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd","Type":"ContainerStarted","Data":"45a1a7fd41be55473393985726ca1d3bc72d474245936ddc9deecba87f58fd1e"}
Jul 28 16:22:36.177057 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.176587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8" event={"ID":"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd","Type":"ContainerStarted","Data":"94983cbdae7f7b0b87b4124346865fb3b423f9d86dcc889b29f62fad6c1caca7"}
Jul 28 16:22:36.177057 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.176609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8" event={"ID":"2dc720a6-f197-4c3d-8ffc-ca0aedd447fd","Type":"ContainerStarted","Data":"ad7b140983b70b6791c9a0a0a0170700f286b5e218a9ca53cbd35d3940a4c3b0"}
Jul 28 16:22:36.178204 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.178171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp" event={"ID":"45cb79a9-7234-499b-93b2-26b628b5b9dd","Type":"ContainerStarted","Data":"4ba69222c498218e85c07e3a90fe7001ff9b0668714a3a12a0fc65ddc1520f4a"}
Jul 28 16:22:36.180153 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.180121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ljfnz" event={"ID":"096be67a-895c-4ebb-a921-30f37159c2dc","Type":"ContainerStarted","Data":"c91c5a26504388252dd198a97b754330420fb63f2cf4ccde055ddf860d416b5f"}
Jul 28 16:22:36.180274 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.180177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ljfnz" event={"ID":"096be67a-895c-4ebb-a921-30f37159c2dc","Type":"ContainerStarted","Data":"1d2850213365730a0330be851733a648a606793ee10ccf6944b411f1453a7833"}
Jul 28 16:22:36.181278 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.181252    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxrnk" event={"ID":"fd850d07-0f15-4189-a81e-46195dc699da","Type":"ContainerStarted","Data":"a548ac52de7ea37d0854db0d78dc3f0c2045d8838ffa4d88a78b91517c250b0c"}
Jul 28 16:22:36.204680 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.204627    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54ff858c4f-ldst8" podStartSLOduration=2.479870415 podStartE2EDuration="7.204609938s" totalImagesPullingTime="4.724739523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:36.203169772 +0000 UTC m=+70.935799450" watchObservedRunningTime="2026-07-28 16:22:36.204609938 +0000 UTC m=+70.937239615"
Jul 28 16:22:36.230247 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.230199    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ljfnz" podStartSLOduration=68.242389437 podStartE2EDuration="1m11.230186533s" totalImagesPullingTime="2.987797096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:36.229913964 +0000 UTC m=+70.962543643" watchObservedRunningTime="2026-07-28 16:22:36.230186533 +0000 UTC m=+70.962816210"
Jul 28 16:22:36.244155 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:36.244113    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5496bc9898-ts2qp" podStartSLOduration=2.520041222 podStartE2EDuration="7.244099048s" totalImagesPullingTime="4.724057826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:36.243291365 +0000 UTC m=+70.975921040" watchObservedRunningTime="2026-07-28 16:22:36.244099048 +0000 UTC m=+70.976728727"
Jul 28 16:22:37.187224 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:37.187183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2nvj5" event={"ID":"5bf00c4a-8cb3-48f4-a3d4-60e55b40d205","Type":"ContainerStarted","Data":"61adc6db8940a2bcccb23ff244130c5ea7617a68f330e0186fb3ae137708d848"}
Jul 28 16:22:37.203358 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:37.203303    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2nvj5" podStartSLOduration=67.569288403 podStartE2EDuration="1m12.20328418s" totalImagesPullingTime="4.633995777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:37.201865798 +0000 UTC m=+71.934495474" watchObservedRunningTime="2026-07-28 16:22:37.20328418 +0000 UTC m=+71.935913858"
Jul 28 16:22:42.210786 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:42.210742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxrnk" event={"ID":"fd850d07-0f15-4189-a81e-46195dc699da","Type":"ContainerStarted","Data":"3792139c64de22dad1be1a7cecbad441d596ebaf2f71cf07a51f122dcec82ed2"}
Jul 28 16:22:44.146422 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.146380    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-22tpm"]
Jul 28 16:22:44.150210 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.150188    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.152841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.152745    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 16:22:44.153347 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.153324    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 16:22:44.153470 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.153323    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 16:22:44.153618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.153601    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bqvfh\""
Jul 28 16:22:44.153705 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.153689    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 16:22:44.154014 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.153978    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 16:22:44.154130 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.154114    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 16:22:44.303956 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.303902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-root\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304148 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.303978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304148 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-sys\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304148 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlmt\" (UniqueName: \"kubernetes.io/projected/bab389b2-348d-4ff4-8686-150bbf487b9f-kube-api-access-qqlmt\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304148 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-metrics-client-ca\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304148 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304393 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-wtmp\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304393 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-textfile\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.304393 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.304297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-tls\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.405742 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-root\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.405742 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.405742 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-sys\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlmt\" (UniqueName: \"kubernetes.io/projected/bab389b2-348d-4ff4-8686-150bbf487b9f-kube-api-access-qqlmt\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-root\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405845    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-sys\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-metrics-client-ca\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-wtmp\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405959    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-textfile\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.405985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-tls\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406456 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.406139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-wtmp\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406456 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.406386    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-metrics-client-ca\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406565 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.406544    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-textfile\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.406754 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.406733    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.409080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.409057    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-tls\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.409080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.409057    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab389b2-348d-4ff4-8686-150bbf487b9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.414558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.414530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlmt\" (UniqueName: \"kubernetes.io/projected/bab389b2-348d-4ff4-8686-150bbf487b9f-kube-api-access-qqlmt\") pod \"node-exporter-22tpm\" (UID: \"bab389b2-348d-4ff4-8686-150bbf487b9f\") " pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.463679 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:44.463649    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-22tpm"
Jul 28 16:22:44.817184 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:22:44.817152    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab389b2_348d_4ff4_8686_150bbf487b9f.slice/crio-e8d0260c5dbbf97b27fda8acc7958a66b6b972dbd5279af55f8e4db50682a139 WatchSource:0}: Error finding container e8d0260c5dbbf97b27fda8acc7958a66b6b972dbd5279af55f8e4db50682a139: Status 404 returned error can't find the container with id e8d0260c5dbbf97b27fda8acc7958a66b6b972dbd5279af55f8e4db50682a139
Jul 28 16:22:45.222960 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:45.222875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxrnk" event={"ID":"fd850d07-0f15-4189-a81e-46195dc699da","Type":"ContainerStarted","Data":"61a7f821ad03040dbc93896e6ea91ad2ab6827483ebf71c8d2d137c73f3a0f7b"}
Jul 28 16:22:45.224278 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:45.224254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-22tpm" event={"ID":"bab389b2-348d-4ff4-8686-150bbf487b9f","Type":"ContainerStarted","Data":"e8d0260c5dbbf97b27fda8acc7958a66b6b972dbd5279af55f8e4db50682a139"}
Jul 28 16:22:46.230051 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:46.229948    2555 generic.go:356] "Generic (PLEG): container finished" podID="bab389b2-348d-4ff4-8686-150bbf487b9f" containerID="6437c5864ce66c143c1e8eec64ea5ffbb5abd9198ee089aa2477fcd6dfd275f5" exitCode=0
Jul 28 16:22:46.230424 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:46.230052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-22tpm" event={"ID":"bab389b2-348d-4ff4-8686-150bbf487b9f","Type":"ContainerDied","Data":"6437c5864ce66c143c1e8eec64ea5ffbb5abd9198ee089aa2477fcd6dfd275f5"}
Jul 28 16:22:47.236864 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:47.236827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-22tpm" event={"ID":"bab389b2-348d-4ff4-8686-150bbf487b9f","Type":"ContainerStarted","Data":"32ef1d22123c41aea7ea0fe6f887a4d44c9d25af24bfe66af94a2258b31f0e32"}
Jul 28 16:22:47.237325 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:47.236928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-22tpm" event={"ID":"bab389b2-348d-4ff4-8686-150bbf487b9f","Type":"ContainerStarted","Data":"794ebac7fd48b8244005e778fa6f55c086606267dd387a5f76b379809d44202f"}
Jul 28 16:22:47.257324 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:47.257265    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-22tpm" podStartSLOduration=2.186376463 podStartE2EDuration="3.257245076s" totalImagesPullingTime="989.981605ms" totalInitContainerRuntime="80.887008ms" isStatefulPod=true podCreationTimestamp="2026-07-28 16:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:47.255561298 +0000 UTC m=+81.988190976" watchObservedRunningTime="2026-07-28 16:22:47.257245076 +0000 UTC m=+81.989874757"
Jul 28 16:22:49.246425 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:49.246390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxrnk" event={"ID":"fd850d07-0f15-4189-a81e-46195dc699da","Type":"ContainerStarted","Data":"4513304034d039aa117a596a5787e71434d7c72c18201c44866d0c1230087a3c"}
Jul 28 16:22:49.264654 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:22:49.264608    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jxrnk" podStartSLOduration=34.287645806 podStartE2EDuration="47.264594966s" totalImagesPullingTime="12.97694916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:22:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:49.263029772 +0000 UTC m=+83.995659452" watchObservedRunningTime="2026-07-28 16:22:49.264594966 +0000 UTC m=+83.997224645"
Jul 28 16:23:04.168343 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:23:04.168309    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgggh"
Jul 28 16:26:25.614840 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:26:25.614804    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:26:25.617391 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:26:25.617365    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:26:25.620823 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:26:25.620803    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 16:27:45.231904 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.231874    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-nsd24"]
Jul 28 16:27:45.234885 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.234871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.237305 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.237280    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 16:27:45.237427 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.237359    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 16:27:45.237427 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.237406    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 16:27:45.237427 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.237412    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vzrc6\""
Jul 28 16:27:45.238380 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.238363    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 16:27:45.238465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.238452    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 16:27:45.244353 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.244332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-nsd24"]
Jul 28 16:27:45.339031 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.338989    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.339126 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.339042    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8tx\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-kube-api-access-vn8tx\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.339126 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.339081    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7825b936-7432-4f83-9455-c4b4505b1d25-cabundle0\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.439939 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.439917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.440080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.439968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8tx\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-kube-api-access-vn8tx\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.440080 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.439994    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7825b936-7432-4f83-9455-c4b4505b1d25-cabundle0\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.440190 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.440086    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 16:27:45.440190 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.440106    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:27:45.440190 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.440116    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:27:45.440190 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.440129    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-nsd24: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:27:45.440190 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.440187    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates podName:7825b936-7432-4f83-9455-c4b4505b1d25 nodeName:}" failed. No retries permitted until 2026-07-28 16:27:45.940167646 +0000 UTC m=+380.672797315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates") pod "keda-operator-77f648d694-nsd24" (UID: "7825b936-7432-4f83-9455-c4b4505b1d25") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:27:45.440633 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.440612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7825b936-7432-4f83-9455-c4b4505b1d25-cabundle0\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.450967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.450945    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8tx\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-kube-api-access-vn8tx\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.943980 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:45.943952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:45.944153 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.944069    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 16:27:45.944153 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.944082    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:27:45.944153 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.944089    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:27:45.944153 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.944098    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-nsd24: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:27:45.944153 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:27:45.944140    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates podName:7825b936-7432-4f83-9455-c4b4505b1d25 nodeName:}" failed. No retries permitted until 2026-07-28 16:27:46.944128551 +0000 UTC m=+381.676758206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates") pod "keda-operator-77f648d694-nsd24" (UID: "7825b936-7432-4f83-9455-c4b4505b1d25") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:27:46.951136 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:46.951102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:46.953539 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:46.953509    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7825b936-7432-4f83-9455-c4b4505b1d25-certificates\") pod \"keda-operator-77f648d694-nsd24\" (UID: \"7825b936-7432-4f83-9455-c4b4505b1d25\") " pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:47.044804 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:47.044777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:47.166435 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:47.166413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-nsd24"]
Jul 28 16:27:47.168729 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:27:47.168689    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7825b936_7432_4f83_9455_c4b4505b1d25.slice/crio-8ced37618d6aaa8f05809b513f61ffcdfd173c079281238d7d9665e839da40fd WatchSource:0}: Error finding container 8ced37618d6aaa8f05809b513f61ffcdfd173c079281238d7d9665e839da40fd: Status 404 returned error can't find the container with id 8ced37618d6aaa8f05809b513f61ffcdfd173c079281238d7d9665e839da40fd
Jul 28 16:27:47.170141 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:47.170125    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:27:47.179751 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:47.179724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-nsd24" event={"ID":"7825b936-7432-4f83-9455-c4b4505b1d25","Type":"ContainerStarted","Data":"8ced37618d6aaa8f05809b513f61ffcdfd173c079281238d7d9665e839da40fd"}
Jul 28 16:27:51.195374 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:51.195343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-nsd24" event={"ID":"7825b936-7432-4f83-9455-c4b4505b1d25","Type":"ContainerStarted","Data":"f4db122d8e4c5dffdccad4ccb7ec01ef5364ffc14a1eaa74d00ece527c32bd47"}
Jul 28 16:27:51.195716 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:51.195410    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:27:51.211643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:27:51.211604    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-nsd24" podStartSLOduration=2.924507284 podStartE2EDuration="6.211590854s" totalImagesPullingTime="3.28708357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:27:51.210019012 +0000 UTC m=+385.942648680" watchObservedRunningTime="2026-07-28 16:27:51.211590854 +0000 UTC m=+385.944220531"
Jul 28 16:28:12.201312 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:28:12.201285    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-nsd24"
Jul 28 16:29:00.784866 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.784793    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:00.786837 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.786815    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:00.789229 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.789210    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 16:29:00.789507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.789495    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 16:29:00.790271 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.790256    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-66kbc\""
Jul 28 16:29:00.790349 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.790331    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 16:29:00.797497 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.797476    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:00.824448 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.824427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrp5\" (UniqueName: \"kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:00.824554 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.824475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:00.925665 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.925640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:00.925749 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.925680    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrp5\" (UniqueName: \"kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:00.925787 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:29:00.925765    2555 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 16:29:00.925832 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:29:00.925823    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert podName:ed5406bd-3334-43e7-b2cf-da5783392e62 nodeName:}" failed. No retries permitted until 2026-07-28 16:29:01.425808934 +0000 UTC m=+456.158438589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert") pod "kserve-controller-manager-85d8b8ddfc-vpnz2" (UID: "ed5406bd-3334-43e7-b2cf-da5783392e62") : secret "kserve-webhook-server-cert" not found
Jul 28 16:29:00.935767 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:00.935744    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrp5\" (UniqueName: \"kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:01.430221 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:01.430198    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:01.432416 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:01.432400    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") pod \"kserve-controller-manager-85d8b8ddfc-vpnz2\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:01.696175 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:01.696124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:01.813967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:01.813940    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:01.818339 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:29:01.818308    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5406bd_3334_43e7_b2cf_da5783392e62.slice/crio-9ac031b7947af87aef1c18b1216ebb826943410947c7340059c01fb5c95d3baa WatchSource:0}: Error finding container 9ac031b7947af87aef1c18b1216ebb826943410947c7340059c01fb5c95d3baa: Status 404 returned error can't find the container with id 9ac031b7947af87aef1c18b1216ebb826943410947c7340059c01fb5c95d3baa
Jul 28 16:29:02.416637 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:02.416599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" event={"ID":"ed5406bd-3334-43e7-b2cf-da5783392e62","Type":"ContainerStarted","Data":"9ac031b7947af87aef1c18b1216ebb826943410947c7340059c01fb5c95d3baa"}
Jul 28 16:29:04.425917 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:04.425875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" event={"ID":"ed5406bd-3334-43e7-b2cf-da5783392e62","Type":"ContainerStarted","Data":"f982bab99fb1943ea8f5001d86bc1a4a97f151eb8e10bd74ca7bccd0e244e7ce"}
Jul 28 16:29:04.426304 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:04.426027    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:04.441806 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:04.441762    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" podStartSLOduration=2.039825073 podStartE2EDuration="4.44174852s" totalImagesPullingTime="2.401923447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:29:04.44081506 +0000 UTC m=+459.173444735" watchObservedRunningTime="2026-07-28 16:29:04.44174852 +0000 UTC m=+459.174378216"
Jul 28 16:29:35.434969 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:35.434939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:43.366174 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.366143    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:43.366635 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.366381    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" podUID="ed5406bd-3334-43e7-b2cf-da5783392e62" containerName="manager" containerID="cri-o://f982bab99fb1943ea8f5001d86bc1a4a97f151eb8e10bd74ca7bccd0e244e7ce" gracePeriod=10
Jul 28 16:29:43.388944 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.388924    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"]
Jul 28 16:29:43.390856 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.390843    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.399237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.399216    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"]
Jul 28 16:29:43.494474 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.494447    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3246ae98-96ac-4ed4-b38c-11450beb213e-cert\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.494577 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.494526    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fls7h\" (UniqueName: \"kubernetes.io/projected/3246ae98-96ac-4ed4-b38c-11450beb213e-kube-api-access-fls7h\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.550808 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.550778    2555 generic.go:356] "Generic (PLEG): container finished" podID="ed5406bd-3334-43e7-b2cf-da5783392e62" containerID="f982bab99fb1943ea8f5001d86bc1a4a97f151eb8e10bd74ca7bccd0e244e7ce" exitCode=0
Jul 28 16:29:43.550932 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.550830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" event={"ID":"ed5406bd-3334-43e7-b2cf-da5783392e62","Type":"ContainerDied","Data":"f982bab99fb1943ea8f5001d86bc1a4a97f151eb8e10bd74ca7bccd0e244e7ce"}
Jul 28 16:29:43.592029 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.591996    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:43.594961 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.594945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3246ae98-96ac-4ed4-b38c-11450beb213e-cert\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.595048 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.594984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fls7h\" (UniqueName: \"kubernetes.io/projected/3246ae98-96ac-4ed4-b38c-11450beb213e-kube-api-access-fls7h\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.597165 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.597138    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3246ae98-96ac-4ed4-b38c-11450beb213e-cert\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.602635 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.602613    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fls7h\" (UniqueName: \"kubernetes.io/projected/3246ae98-96ac-4ed4-b38c-11450beb213e-kube-api-access-fls7h\") pod \"kserve-controller-manager-85d8b8ddfc-f9w9s\" (UID: \"3246ae98-96ac-4ed4-b38c-11450beb213e\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.696281 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.696196    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhrp5\" (UniqueName: \"kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5\") pod \"ed5406bd-3334-43e7-b2cf-da5783392e62\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") "
Jul 28 16:29:43.696281 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.696242    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") pod \"ed5406bd-3334-43e7-b2cf-da5783392e62\" (UID: \"ed5406bd-3334-43e7-b2cf-da5783392e62\") "
Jul 28 16:29:43.698441 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.698405    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5" (OuterVolumeSpecName: "kube-api-access-hhrp5") pod "ed5406bd-3334-43e7-b2cf-da5783392e62" (UID: "ed5406bd-3334-43e7-b2cf-da5783392e62"). InnerVolumeSpecName "kube-api-access-hhrp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:29:43.698539 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.698440    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert" (OuterVolumeSpecName: "cert") pod "ed5406bd-3334-43e7-b2cf-da5783392e62" (UID: "ed5406bd-3334-43e7-b2cf-da5783392e62"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:29:43.735621 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.735597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:43.797573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.797548    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrp5\" (UniqueName: \"kubernetes.io/projected/ed5406bd-3334-43e7-b2cf-da5783392e62-kube-api-access-hhrp5\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:29:43.797573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.797572    2555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed5406bd-3334-43e7-b2cf-da5783392e62-cert\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:29:43.852849 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:43.852785    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"]
Jul 28 16:29:43.855205 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:29:43.855183    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3246ae98_96ac_4ed4_b38c_11450beb213e.slice/crio-2a0ac4409ab4b3e465ce909d1acfd4ccf51cf5380e4ab6228042f69d50bb823a WatchSource:0}: Error finding container 2a0ac4409ab4b3e465ce909d1acfd4ccf51cf5380e4ab6228042f69d50bb823a: Status 404 returned error can't find the container with id 2a0ac4409ab4b3e465ce909d1acfd4ccf51cf5380e4ab6228042f69d50bb823a
Jul 28 16:29:44.554838 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.554813    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"
Jul 28 16:29:44.554838 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.554820    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-vpnz2" event={"ID":"ed5406bd-3334-43e7-b2cf-da5783392e62","Type":"ContainerDied","Data":"9ac031b7947af87aef1c18b1216ebb826943410947c7340059c01fb5c95d3baa"}
Jul 28 16:29:44.555327 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.554902    2555 scope.go:117] "RemoveContainer" containerID="f982bab99fb1943ea8f5001d86bc1a4a97f151eb8e10bd74ca7bccd0e244e7ce"
Jul 28 16:29:44.556465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.556427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s" event={"ID":"3246ae98-96ac-4ed4-b38c-11450beb213e","Type":"ContainerStarted","Data":"c590d608b0edab41553b778d5d5b57cf74e8ece0a1189ebb1b5cc11ddd88c2d6"}
Jul 28 16:29:44.556654 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.556578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s" event={"ID":"3246ae98-96ac-4ed4-b38c-11450beb213e","Type":"ContainerStarted","Data":"2a0ac4409ab4b3e465ce909d1acfd4ccf51cf5380e4ab6228042f69d50bb823a"}
Jul 28 16:29:44.556654 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.556670    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:29:44.574763 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.574729    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s" podStartSLOduration=1.094612421 podStartE2EDuration="1.574716043s" totalImagesPullingTime="480.103622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:29:44.574332902 +0000 UTC m=+499.306962579" watchObservedRunningTime="2026-07-28 16:29:44.574716043 +0000 UTC m=+499.307345777"
Jul 28 16:29:44.587303 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.587259    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:44.592356 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:44.592339    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-vpnz2"]
Jul 28 16:29:45.823834 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:29:45.823802    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5406bd-3334-43e7-b2cf-da5783392e62" path="/var/lib/kubelet/pods/ed5406bd-3334-43e7-b2cf-da5783392e62/volumes"
Jul 28 16:30:15.566124 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:15.566092    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-f9w9s"
Jul 28 16:30:16.488388 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.488353    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-5mq4z"]
Jul 28 16:30:16.488607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.488596    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5406bd-3334-43e7-b2cf-da5783392e62" containerName="manager"
Jul 28 16:30:16.488648 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.488610    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5406bd-3334-43e7-b2cf-da5783392e62" containerName="manager"
Jul 28 16:30:16.488683 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.488649    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5406bd-3334-43e7-b2cf-da5783392e62" containerName="manager"
Jul 28 16:30:16.491232 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.491214    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:16.493604 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.493587    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 16:30:16.493706 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.493608    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-l2mhv\""
Jul 28 16:30:16.501956 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.501935    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-5mq4z"]
Jul 28 16:30:16.605785 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.605760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:16.606107 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.605793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xsl\" (UniqueName: \"kubernetes.io/projected/1af2fc51-9647-48df-864f-252b55ef86b9-kube-api-access-l9xsl\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:16.706884 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.706858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:16.706986 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.706901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xsl\" (UniqueName: \"kubernetes.io/projected/1af2fc51-9647-48df-864f-252b55ef86b9-kube-api-access-l9xsl\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:16.707078 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:30:16.706996    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 16:30:16.707133 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:30:16.707082    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert podName:1af2fc51-9647-48df-864f-252b55ef86b9 nodeName:}" failed. No retries permitted until 2026-07-28 16:30:17.207059659 +0000 UTC m=+531.939689323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert") pod "odh-model-controller-75d9cdbddf-5mq4z" (UID: "1af2fc51-9647-48df-864f-252b55ef86b9") : secret "odh-model-controller-webhook-cert" not found
Jul 28 16:30:16.717192 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:16.717173    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xsl\" (UniqueName: \"kubernetes.io/projected/1af2fc51-9647-48df-864f-252b55ef86b9-kube-api-access-l9xsl\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:17.209697 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:17.209672    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:17.211919 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:17.211896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af2fc51-9647-48df-864f-252b55ef86b9-cert\") pod \"odh-model-controller-75d9cdbddf-5mq4z\" (UID: \"1af2fc51-9647-48df-864f-252b55ef86b9\") " pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:17.400298 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:17.400273    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:17.521249 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:17.521221    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-5mq4z"]
Jul 28 16:30:17.524256 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:30:17.524220    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af2fc51_9647_48df_864f_252b55ef86b9.slice/crio-781c906e604fa60d5d7b22ae775b59133f8f71bfdbb50843480903021655daa6 WatchSource:0}: Error finding container 781c906e604fa60d5d7b22ae775b59133f8f71bfdbb50843480903021655daa6: Status 404 returned error can't find the container with id 781c906e604fa60d5d7b22ae775b59133f8f71bfdbb50843480903021655daa6
Jul 28 16:30:17.661872 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:17.661841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z" event={"ID":"1af2fc51-9647-48df-864f-252b55ef86b9","Type":"ContainerStarted","Data":"781c906e604fa60d5d7b22ae775b59133f8f71bfdbb50843480903021655daa6"}
Jul 28 16:30:20.674995 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:20.674902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z" event={"ID":"1af2fc51-9647-48df-864f-252b55ef86b9","Type":"ContainerStarted","Data":"0cf256ed6d0ec2d5eeabab22f52d5142a9e70079ebc5befd65d43df3e626f96d"}
Jul 28 16:30:20.675391 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:20.675107    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:30:20.690483 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:20.690444    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z" podStartSLOduration=1.955667007 podStartE2EDuration="4.690429372s" totalImagesPullingTime="2.734762365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:20.689990966 +0000 UTC m=+535.422620641" watchObservedRunningTime="2026-07-28 16:30:20.690429372 +0000 UTC m=+535.423059048"
Jul 28 16:30:31.680797 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:30:31.680763    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-5mq4z"
Jul 28 16:31:25.637136 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:31:25.637110    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:31:25.637618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:31:25.637556    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:32:07.067500 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.067469    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:32:07.072669 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.072639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.075415 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.075391    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:32:07.076108 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.076088    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zwq7z\""
Jul 28 16:32:07.076188 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.076116    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:32:07.076237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.076168    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2ad5-kube-rbac-proxy-sar-config\""
Jul 28 16:32:07.076337 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.076322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2ad5-predictor-serving-cert\""
Jul 28 16:32:07.081152 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.081129    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:32:07.142326 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.142298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.142439 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.142331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxrd\" (UniqueName: \"kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.142439 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.142365    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.242891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.242862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.243046 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.242904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.243113 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.243051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxrd\" (UniqueName: \"kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.243442 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.243424    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.245268 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.245250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.253324 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.253069    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxrd\" (UniqueName: \"kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd\") pod \"error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") " pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.276608 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.276581    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:32:07.279106 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.279089    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.281733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.281615    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 28 16:32:07.281733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.281678    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:32:07.281733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.281702    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 28 16:32:07.281733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.281623    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:32:07.290442 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.290424    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:32:07.343434 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.343378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdb8\" (UniqueName: \"kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.343434 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.343408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.343559 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.343438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.343559 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.343507    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.343559 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.343538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.384440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.384415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:07.444573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.444547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.444682 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.444584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.444682 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.444631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdb8\" (UniqueName: \"kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.444682 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.444657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.444782 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.444690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.444782 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:32:07.444714    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 16:32:07.444842 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:32:07.444787    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls podName:e8a1a80a-9dcb-4596-be10-bf900623d462 nodeName:}" failed. No retries permitted until 2026-07-28 16:32:07.944767084 +0000 UTC m=+642.677396740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls") pod "isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 16:32:07.445476 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.445319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.445476 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.445434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.445897 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.445850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.454990 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.454856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdb8\" (UniqueName: \"kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.511741 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.511709    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:32:07.515546 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:32:07.515521    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22c69c5_9349_401e_ad26_b1937155ce32.slice/crio-38ffb57bcaabc0e29c994b5af653aeeef1116733aaa2914ae910e83814c907f3 WatchSource:0}: Error finding container 38ffb57bcaabc0e29c994b5af653aeeef1116733aaa2914ae910e83814c907f3: Status 404 returned error can't find the container with id 38ffb57bcaabc0e29c994b5af653aeeef1116733aaa2914ae910e83814c907f3
Jul 28 16:32:07.949041 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.949010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:07.951508 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:07.951432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:08.013573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:08.013542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerStarted","Data":"38ffb57bcaabc0e29c994b5af653aeeef1116733aaa2914ae910e83814c907f3"}
Jul 28 16:32:08.189878 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:08.189845    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:08.376757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:08.376626    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:32:08.381779 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:32:08.381749    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a1a80a_9dcb_4596_be10_bf900623d462.slice/crio-6b6a82a3be0d7bbeafbffdddc8da4dd502bc3eb27fa33958ed0b35ae871c2c9f WatchSource:0}: Error finding container 6b6a82a3be0d7bbeafbffdddc8da4dd502bc3eb27fa33958ed0b35ae871c2c9f: Status 404 returned error can't find the container with id 6b6a82a3be0d7bbeafbffdddc8da4dd502bc3eb27fa33958ed0b35ae871c2c9f
Jul 28 16:32:09.019955 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:09.019911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerStarted","Data":"6b6a82a3be0d7bbeafbffdddc8da4dd502bc3eb27fa33958ed0b35ae871c2c9f"}
Jul 28 16:32:21.071654 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:21.071597    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerStarted","Data":"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"}
Jul 28 16:32:21.073440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:21.073407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerStarted","Data":"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"}
Jul 28 16:32:23.083695 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:23.083661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerStarted","Data":"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"}
Jul 28 16:32:23.084228 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:23.083850    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:23.084228 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:23.083893    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:23.085152 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:23.085124    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:32:23.101447 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:23.101409    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podStartSLOduration=1.102862789 podStartE2EDuration="16.101396043s" totalImagesPullingTime="14.998533254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:23.099315375 +0000 UTC m=+657.831945051" watchObservedRunningTime="2026-07-28 16:32:23.101396043 +0000 UTC m=+657.834025719"
Jul 28 16:32:24.087078 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:24.087037    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:32:25.092469 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:25.092437    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerID="8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d" exitCode=0
Jul 28 16:32:25.092816 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:25.092510    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerDied","Data":"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"}
Jul 28 16:32:29.092101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:29.092023    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:32:29.093146 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:29.093057    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:32:32.128049 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.127926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerStarted","Data":"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"}
Jul 28 16:32:32.128472 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.128094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerStarted","Data":"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"}
Jul 28 16:32:32.128472 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.128419    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:32.128597 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.128577    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:32.129733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.129708    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:32:32.146886 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:32.146836    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podStartSLOduration=2.771174811 podStartE2EDuration="25.14681995s" totalImagesPullingTime="18.480303767s" totalInitContainerRuntime="3.895341372s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:32.144716359 +0000 UTC m=+666.877346035" watchObservedRunningTime="2026-07-28 16:32:32.14681995 +0000 UTC m=+666.879449628"
Jul 28 16:32:33.131602 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:33.131558    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:32:38.135793 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:38.135762    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:32:38.136307 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:38.136284    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:32:39.093362 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:39.093325    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:32:48.136281 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:48.136235    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:32:49.092815 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:49.092773    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:32:58.136242 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:58.136199    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:32:59.092589 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:32:59.092547    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:33:08.136993 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:08.136949    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:33:09.092961 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:09.092933    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:33:18.136961 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:18.136883    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:33:26.683728 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.683690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:33:26.686108 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.686088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:26.688428 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.688407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2ad5-serving-cert\""
Jul 28 16:33:26.688542 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.688409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2ad5-kube-rbac-proxy-sar-config\""
Jul 28 16:33:26.697078 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.697049    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:33:26.758903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.758869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:26.758903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.758909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:26.860083 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.860047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:26.860253 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.860159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:26.860253 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:33:26.860209    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b2ad5-serving-cert: secret "switch-graph-b2ad5-serving-cert" not found
Jul 28 16:33:26.860323 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:33:26.860283    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls podName:5d35afce-f447-4c44-a61f-38d89aa16812 nodeName:}" failed. No retries permitted until 2026-07-28 16:33:27.360261474 +0000 UTC m=+722.092891130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls") pod "switch-graph-b2ad5-6f6d6b5ff5-p57bx" (UID: "5d35afce-f447-4c44-a61f-38d89aa16812") : secret "switch-graph-b2ad5-serving-cert" not found
Jul 28 16:33:26.860845 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:26.860828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:27.363858 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:27.363823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:27.366392 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:27.366359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") pod \"switch-graph-b2ad5-6f6d6b5ff5-p57bx\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") " pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:27.596477 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:27.596431    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:27.731041 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:27.730996    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:33:27.733442 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:33:27.733413    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35afce_f447_4c44_a61f_38d89aa16812.slice/crio-752e6264cf484de3f2656d299f685d3c6d3cdedc11b0e48b473d7b0914299523 WatchSource:0}: Error finding container 752e6264cf484de3f2656d299f685d3c6d3cdedc11b0e48b473d7b0914299523: Status 404 returned error can't find the container with id 752e6264cf484de3f2656d299f685d3c6d3cdedc11b0e48b473d7b0914299523
Jul 28 16:33:27.735371 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:27.735354    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:33:28.136827 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:28.136787    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:33:28.332695 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:28.332659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" event={"ID":"5d35afce-f447-4c44-a61f-38d89aa16812","Type":"ContainerStarted","Data":"752e6264cf484de3f2656d299f685d3c6d3cdedc11b0e48b473d7b0914299523"}
Jul 28 16:33:30.341678 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:30.341639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" event={"ID":"5d35afce-f447-4c44-a61f-38d89aa16812","Type":"ContainerStarted","Data":"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"}
Jul 28 16:33:30.342171 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:30.341798    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:30.357786 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:30.357722    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podStartSLOduration=2.218281968 podStartE2EDuration="4.357704653s" totalImagesPullingTime="2.139422685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:33:30.356357241 +0000 UTC m=+725.088986920" watchObservedRunningTime="2026-07-28 16:33:30.357704653 +0000 UTC m=+725.090334334"
Jul 28 16:33:36.350865 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.350831    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:36.798086 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.798050    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:33:36.798295 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.798267    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" containerID="cri-o://423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d" gracePeriod=30
Jul 28 16:33:36.954621 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.954590    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:33:36.954885 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.954859    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" containerID="cri-o://faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb" gracePeriod=30
Jul 28 16:33:36.955095 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:36.954940    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kube-rbac-proxy" containerID="cri-o://cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae" gracePeriod=30
Jul 28 16:33:37.065785 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.065672    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:33:37.068078 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.068058    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.070465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.070442    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ba9d-predictor-serving-cert\""
Jul 28 16:33:37.070465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.070456    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ba9d-kube-rbac-proxy-sar-config\""
Jul 28 16:33:37.080156 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.080128    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:33:37.243914 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.243880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.244110 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.244014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.244110 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.244070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh765\" (UniqueName: \"kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.345045 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.344926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh765\" (UniqueName: \"kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.345045 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.344973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.345045 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.345041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.345780 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.345754    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.347733 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.347702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.354608 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.354580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh765\" (UniqueName: \"kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765\") pod \"error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") " pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.376161 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.376126    2555 generic.go:356] "Generic (PLEG): container finished" podID="a22c69c5-9349-401e-ad26-b1937155ce32" containerID="cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae" exitCode=2
Jul 28 16:33:37.376316 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.376182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerDied","Data":"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"}
Jul 28 16:33:37.379399 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.379380    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:37.508212 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:37.508056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:33:37.510791 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:33:37.510768    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a14f67c_97b7_4857_a8f1_61bae732ad19.slice/crio-862d43e1edb49b213c0bae2e8b8c554d4cf87b6a14e0813f490f8783e1663c09 WatchSource:0}: Error finding container 862d43e1edb49b213c0bae2e8b8c554d4cf87b6a14e0813f490f8783e1663c09: Status 404 returned error can't find the container with id 862d43e1edb49b213c0bae2e8b8c554d4cf87b6a14e0813f490f8783e1663c09
Jul 28 16:33:38.136465 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.136428    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:33:38.382174 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.382133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerStarted","Data":"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"}
Jul 28 16:33:38.382595 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.382202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerStarted","Data":"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"}
Jul 28 16:33:38.382595 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.382223    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerStarted","Data":"862d43e1edb49b213c0bae2e8b8c554d4cf87b6a14e0813f490f8783e1663c09"}
Jul 28 16:33:38.382595 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.382300    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:38.400978 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:38.400870    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podStartSLOduration=1.400854121 podStartE2EDuration="1.400854121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:33:38.399980546 +0000 UTC m=+733.132610237" watchObservedRunningTime="2026-07-28 16:33:38.400854121 +0000 UTC m=+733.133483799"
Jul 28 16:33:39.088352 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:39.088306    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 28 16:33:39.092751 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:39.092725    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 16:33:39.386865 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:39.386782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:39.388181 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:39.388152    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:33:40.204508 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.204485    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:33:40.369738 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.369661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config\") pod \"a22c69c5-9349-401e-ad26-b1937155ce32\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") "
Jul 28 16:33:40.369880 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.369749    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls\") pod \"a22c69c5-9349-401e-ad26-b1937155ce32\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") "
Jul 28 16:33:40.369880 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.369769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwxrd\" (UniqueName: \"kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd\") pod \"a22c69c5-9349-401e-ad26-b1937155ce32\" (UID: \"a22c69c5-9349-401e-ad26-b1937155ce32\") "
Jul 28 16:33:40.370032 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.369987    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a22c69c5-9349-401e-ad26-b1937155ce32" (UID: "a22c69c5-9349-401e-ad26-b1937155ce32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:40.372094 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.372067    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a22c69c5-9349-401e-ad26-b1937155ce32" (UID: "a22c69c5-9349-401e-ad26-b1937155ce32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:33:40.372199 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.372106    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd" (OuterVolumeSpecName: "kube-api-access-cwxrd") pod "a22c69c5-9349-401e-ad26-b1937155ce32" (UID: "a22c69c5-9349-401e-ad26-b1937155ce32"). InnerVolumeSpecName "kube-api-access-cwxrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:33:40.392139 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392113    2555 generic.go:356] "Generic (PLEG): container finished" podID="a22c69c5-9349-401e-ad26-b1937155ce32" containerID="faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb" exitCode=0
Jul 28 16:33:40.392431 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392160    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"
Jul 28 16:33:40.392431 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerDied","Data":"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"}
Jul 28 16:33:40.392431 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6" event={"ID":"a22c69c5-9349-401e-ad26-b1937155ce32","Type":"ContainerDied","Data":"38ffb57bcaabc0e29c994b5af653aeeef1116733aaa2914ae910e83814c907f3"}
Jul 28 16:33:40.392431 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392276    2555 scope.go:117] "RemoveContainer" containerID="cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"
Jul 28 16:33:40.392745 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.392708    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:33:40.400065 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.400048    2555 scope.go:117] "RemoveContainer" containerID="faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"
Jul 28 16:33:40.406953 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.406938    2555 scope.go:117] "RemoveContainer" containerID="cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"
Jul 28 16:33:40.407199 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:33:40.407183    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae\": container with ID starting with cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae not found: ID does not exist" containerID="cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"
Jul 28 16:33:40.407243 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.407206    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae"} err="failed to get container status \"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae\": rpc error: code = NotFound desc = could not find container \"cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae\": container with ID starting with cef76e5f74839ee4de9e60ab3b79e6aed65aab20ec99317632a61aeb540308ae not found: ID does not exist"
Jul 28 16:33:40.407243 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.407234    2555 scope.go:117] "RemoveContainer" containerID="faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"
Jul 28 16:33:40.407458 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:33:40.407442    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb\": container with ID starting with faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb not found: ID does not exist" containerID="faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"
Jul 28 16:33:40.407506 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.407464    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb"} err="failed to get container status \"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb\": rpc error: code = NotFound desc = could not find container \"faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb\": container with ID starting with faac2b0ba424103edfb04a258acf29a82083f10b23d6343c3eb8ec0de7e16adb not found: ID does not exist"
Jul 28 16:33:40.411726 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.411703    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:33:40.415281 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.415262    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2ad5-predictor-c8c4fb8c4-xbvl6"]
Jul 28 16:33:40.470287 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.470253    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a22c69c5-9349-401e-ad26-b1937155ce32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:33:40.470287 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.470283    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a22c69c5-9349-401e-ad26-b1937155ce32-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:33:40.470470 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:40.470297    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxrd\" (UniqueName: \"kubernetes.io/projected/a22c69c5-9349-401e-ad26-b1937155ce32-kube-api-access-cwxrd\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:33:41.349871 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:41.349832    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:33:41.823185 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:41.823148    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" path="/var/lib/kubelet/pods/a22c69c5-9349-401e-ad26-b1937155ce32/volumes"
Jul 28 16:33:45.397050 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:45.397022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:33:45.397671 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:45.397642    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:33:46.349773 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:46.349738    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:33:48.137165 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:48.137138    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:33:51.349655 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:51.349618    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:33:51.350129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:51.349760    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:33:55.397799 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:55.397759    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:33:56.349829 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:33:56.349790    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:01.349628 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:01.349588    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:05.397894 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:05.397856    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:34:06.349518 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.349475    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:06.943556 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.943532    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:34:06.957190 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.957164    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") pod \"5d35afce-f447-4c44-a61f-38d89aa16812\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") "
Jul 28 16:34:06.957314 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.957214    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle\") pod \"5d35afce-f447-4c44-a61f-38d89aa16812\" (UID: \"5d35afce-f447-4c44-a61f-38d89aa16812\") "
Jul 28 16:34:06.957564 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.957542    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5d35afce-f447-4c44-a61f-38d89aa16812" (UID: "5d35afce-f447-4c44-a61f-38d89aa16812"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:34:06.959455 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:06.959427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d35afce-f447-4c44-a61f-38d89aa16812" (UID: "5d35afce-f447-4c44-a61f-38d89aa16812"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:34:07.057818 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.057747    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d35afce-f447-4c44-a61f-38d89aa16812-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:07.057818 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.057772    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d35afce-f447-4c44-a61f-38d89aa16812-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:07.490604 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.490516    2555 generic.go:356] "Generic (PLEG): container finished" podID="5d35afce-f447-4c44-a61f-38d89aa16812" containerID="423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d" exitCode=0
Jul 28 16:34:07.490604 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.490574    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"
Jul 28 16:34:07.490809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.490607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" event={"ID":"5d35afce-f447-4c44-a61f-38d89aa16812","Type":"ContainerDied","Data":"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"}
Jul 28 16:34:07.490809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.490677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx" event={"ID":"5d35afce-f447-4c44-a61f-38d89aa16812","Type":"ContainerDied","Data":"752e6264cf484de3f2656d299f685d3c6d3cdedc11b0e48b473d7b0914299523"}
Jul 28 16:34:07.490809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.490698    2555 scope.go:117] "RemoveContainer" containerID="423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"
Jul 28 16:34:07.499279 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.499265    2555 scope.go:117] "RemoveContainer" containerID="423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"
Jul 28 16:34:07.499522 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:07.499503    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d\": container with ID starting with 423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d not found: ID does not exist" containerID="423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"
Jul 28 16:34:07.499569 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.499530    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d"} err="failed to get container status \"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d\": rpc error: code = NotFound desc = could not find container \"423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d\": container with ID starting with 423e781db9353cb0c6b2fd71222bf660f5b781f4cf151bfb703e46f95579fe7d not found: ID does not exist"
Jul 28 16:34:07.509670 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.509651    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:34:07.513406 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.513386    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2ad5-6f6d6b5ff5-p57bx"]
Jul 28 16:34:07.823651 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:07.823618    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" path="/var/lib/kubelet/pods/5d35afce-f447-4c44-a61f-38d89aa16812/volumes"
Jul 28 16:34:15.398043 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:15.397980    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:34:25.398147 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:25.398116    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:34:26.723567 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723534    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723804    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723816    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723826    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723831    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723838    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kube-rbac-proxy"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723844    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kube-rbac-proxy"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723890    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kserve-container"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22c69c5-9349-401e-ad26-b1937155ce32" containerName="kube-rbac-proxy"
Jul 28 16:34:26.723962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.723904    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d35afce-f447-4c44-a61f-38d89aa16812" containerName="switch-graph-b2ad5"
Jul 28 16:34:26.727971 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.727955    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.730107 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.730087    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 16:34:26.730195 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.730094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 16:34:26.736139 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.736120    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:34:26.795336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.795303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.795449 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.795356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.896510 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.896477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.896657 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.896524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.897244 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.897186    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:26.899052 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:26.899031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls\") pod \"model-chainer-c5b4df566-hptsh\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") " pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:27.039448 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.039425    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:27.162572 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.162542    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:34:27.165274 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:34:27.165246    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfe2578_34f0_4b86_8d74_5045ab27a60f.slice/crio-07743da5c7b46e9de72a8bb54add96450988526a5b80ec66741b0940042a237a WatchSource:0}: Error finding container 07743da5c7b46e9de72a8bb54add96450988526a5b80ec66741b0940042a237a: Status 404 returned error can't find the container with id 07743da5c7b46e9de72a8bb54add96450988526a5b80ec66741b0940042a237a
Jul 28 16:34:27.566327 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.566297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" event={"ID":"0bfe2578-34f0-4b86-8d74-5045ab27a60f","Type":"ContainerStarted","Data":"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"}
Jul 28 16:34:27.566489 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.566370    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" event={"ID":"0bfe2578-34f0-4b86-8d74-5045ab27a60f","Type":"ContainerStarted","Data":"07743da5c7b46e9de72a8bb54add96450988526a5b80ec66741b0940042a237a"}
Jul 28 16:34:27.566489 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.566391    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:27.582161 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:27.582120    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podStartSLOduration=1.582105981 podStartE2EDuration="1.582105981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:34:27.581300741 +0000 UTC m=+782.313930419" watchObservedRunningTime="2026-07-28 16:34:27.582105981 +0000 UTC m=+782.314735659"
Jul 28 16:34:33.575840 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:33.575811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:36.824534 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:36.824499    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:34:36.824958 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:36.824718    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" containerID="cri-o://d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161" gracePeriod=30
Jul 28 16:34:36.953254 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:36.953210    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:34:36.953574 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:36.953510    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" containerID="cri-o://ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65" gracePeriod=30
Jul 28 16:34:36.953691 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:36.953540    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kube-rbac-proxy" containerID="cri-o://b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d" gracePeriod=30
Jul 28 16:34:37.028989 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.028926    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:34:37.032586 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.032565    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.035035 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.034995    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7ba9d-serving-cert\""
Jul 28 16:34:37.035262 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.035240    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7ba9d-kube-rbac-proxy-sar-config\""
Jul 28 16:34:37.047824 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.047799    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:34:37.066874 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.066839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.067135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.066893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.167787 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.167706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.167787 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.167744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.167957 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:37.167862    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-7ba9d-serving-cert: secret "switch-graph-7ba9d-serving-cert" not found
Jul 28 16:34:37.167957 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:37.167939    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls podName:b1e6379f-d775-4c57-bd47-d9e00c96f420 nodeName:}" failed. No retries permitted until 2026-07-28 16:34:37.667920874 +0000 UTC m=+792.400550534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls") pod "switch-graph-7ba9d-5757fd8cc-68rtk" (UID: "b1e6379f-d775-4c57-bd47-d9e00c96f420") : secret "switch-graph-7ba9d-serving-cert" not found
Jul 28 16:34:37.168346 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.168330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.224441 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.224410    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:34:37.227768 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.227749    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.230133 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.230114    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2b606-kube-rbac-proxy-sar-config\""
Jul 28 16:34:37.230309 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.230297    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2b606-predictor-serving-cert\""
Jul 28 16:34:37.237375 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.237351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:34:37.268730 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.268691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.268846 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.268815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.268888 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.268856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtng\" (UniqueName: \"kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.370298 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.370261    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.370478 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.370309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtng\" (UniqueName: \"kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.370478 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.370365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.370584 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:37.370491    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2b606-predictor-serving-cert: secret "error-404-isvc-2b606-predictor-serving-cert" not found
Jul 28 16:34:37.370584 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:37.370555    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls podName:6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0 nodeName:}" failed. No retries permitted until 2026-07-28 16:34:37.870537845 +0000 UTC m=+792.603167524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls") pod "error-404-isvc-2b606-predictor-76cf8477c8-wnb86" (UID: "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0") : secret "error-404-isvc-2b606-predictor-serving-cert" not found
Jul 28 16:34:37.370959 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.370941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.378835 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.378810    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtng\" (UniqueName: \"kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.606643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.606611    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerID="b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d" exitCode=2
Jul 28 16:34:37.606812 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.606664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerDied","Data":"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"}
Jul 28 16:34:37.673507 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.673478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.675938 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.675911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") pod \"switch-graph-7ba9d-5757fd8cc-68rtk\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") " pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:37.875387 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.875309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.877749 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.877727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") pod \"error-404-isvc-2b606-predictor-76cf8477c8-wnb86\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") " pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:37.945470 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:37.945440    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:38.069881 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.069855    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:34:38.072250 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:34:38.072211    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e6379f_d775_4c57_bd47_d9e00c96f420.slice/crio-4d87db0102a8c819a9f5a2fcebab2e3c7764129156c8a8487ea72b8aef24bfb8 WatchSource:0}: Error finding container 4d87db0102a8c819a9f5a2fcebab2e3c7764129156c8a8487ea72b8aef24bfb8: Status 404 returned error can't find the container with id 4d87db0102a8c819a9f5a2fcebab2e3c7764129156c8a8487ea72b8aef24bfb8
Jul 28 16:34:38.134637 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.134604    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:34:38.136462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.136436    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:34:38.137579 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.137558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:38.266359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.266332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:34:38.268478 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:34:38.268452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec7b3f7_6eae_42e8_9e8c_364b14e5eed0.slice/crio-61c6f427db7a19cc8480c247bb3efa217b831878b53db2110ab2cda8ce0bc2e9 WatchSource:0}: Error finding container 61c6f427db7a19cc8480c247bb3efa217b831878b53db2110ab2cda8ce0bc2e9: Status 404 returned error can't find the container with id 61c6f427db7a19cc8480c247bb3efa217b831878b53db2110ab2cda8ce0bc2e9
Jul 28 16:34:38.573935 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.573902    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:38.611676 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.611644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerStarted","Data":"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"}
Jul 28 16:34:38.611810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.611709    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerStarted","Data":"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"}
Jul 28 16:34:38.611810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.611737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerStarted","Data":"61c6f427db7a19cc8480c247bb3efa217b831878b53db2110ab2cda8ce0bc2e9"}
Jul 28 16:34:38.611810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.611771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:38.611810 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.611800    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:38.613298 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.613271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" event={"ID":"b1e6379f-d775-4c57-bd47-d9e00c96f420","Type":"ContainerStarted","Data":"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"}
Jul 28 16:34:38.613384 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.613315    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:38.613384 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.613331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" event={"ID":"b1e6379f-d775-4c57-bd47-d9e00c96f420","Type":"ContainerStarted","Data":"4d87db0102a8c819a9f5a2fcebab2e3c7764129156c8a8487ea72b8aef24bfb8"}
Jul 28 16:34:38.613455 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.613419    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:34:38.629884 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.629797    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podStartSLOduration=1.6297846599999999 podStartE2EDuration="1.62978466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:34:38.628736533 +0000 UTC m=+793.361366211" watchObservedRunningTime="2026-07-28 16:34:38.62978466 +0000 UTC m=+793.362414337"
Jul 28 16:34:38.642832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:38.642793    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podStartSLOduration=1.6427817569999998 podStartE2EDuration="1.642781757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:34:38.642316887 +0000 UTC m=+793.374946566" watchObservedRunningTime="2026-07-28 16:34:38.642781757 +0000 UTC m=+793.375411434"
Jul 28 16:34:39.617091 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:39.617049    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:34:41.500914 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.500890    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:34:41.605265 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605174    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") pod \"e8a1a80a-9dcb-4596-be10-bf900623d462\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") "
Jul 28 16:34:41.605265 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605221    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert\") pod \"e8a1a80a-9dcb-4596-be10-bf900623d462\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") "
Jul 28 16:34:41.605496 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605274    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvdb8\" (UniqueName: \"kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8\") pod \"e8a1a80a-9dcb-4596-be10-bf900623d462\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") "
Jul 28 16:34:41.605496 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605291    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config\") pod \"e8a1a80a-9dcb-4596-be10-bf900623d462\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") "
Jul 28 16:34:41.605496 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605309    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location\") pod \"e8a1a80a-9dcb-4596-be10-bf900623d462\" (UID: \"e8a1a80a-9dcb-4596-be10-bf900623d462\") "
Jul 28 16:34:41.605660 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605630    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8a1a80a-9dcb-4596-be10-bf900623d462" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:34:41.605767 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605657    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8a1a80a-9dcb-4596-be10-bf900623d462" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:34:41.605767 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.605668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8a1a80a-9dcb-4596-be10-bf900623d462" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:34:41.607429 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.607411    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8a1a80a-9dcb-4596-be10-bf900623d462" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:34:41.607484 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.607427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8" (OuterVolumeSpecName: "kube-api-access-rvdb8") pod "e8a1a80a-9dcb-4596-be10-bf900623d462" (UID: "e8a1a80a-9dcb-4596-be10-bf900623d462"). InnerVolumeSpecName "kube-api-access-rvdb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:34:41.625225 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.625199    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerID="ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65" exitCode=0
Jul 28 16:34:41.625317 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.625256    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"
Jul 28 16:34:41.625317 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.625280    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerDied","Data":"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"}
Jul 28 16:34:41.625400 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.625339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb" event={"ID":"e8a1a80a-9dcb-4596-be10-bf900623d462","Type":"ContainerDied","Data":"6b6a82a3be0d7bbeafbffdddc8da4dd502bc3eb27fa33958ed0b35ae871c2c9f"}
Jul 28 16:34:41.625400 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.625368    2555 scope.go:117] "RemoveContainer" containerID="b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"
Jul 28 16:34:41.636025 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.634154    2555 scope.go:117] "RemoveContainer" containerID="ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"
Jul 28 16:34:41.645983 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.645971    2555 scope.go:117] "RemoveContainer" containerID="8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"
Jul 28 16:34:41.647391 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.647364    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:34:41.650962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.650941    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-z6fkb"]
Jul 28 16:34:41.653891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.653874    2555 scope.go:117] "RemoveContainer" containerID="b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"
Jul 28 16:34:41.654160 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:41.654137    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d\": container with ID starting with b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d not found: ID does not exist" containerID="b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"
Jul 28 16:34:41.654249 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.654171    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d"} err="failed to get container status \"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d\": rpc error: code = NotFound desc = could not find container \"b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d\": container with ID starting with b4f9aefda13355afdca94d76aa93a5243cdcf4a51f9fb5ec7bcb2f2af4e0a70d not found: ID does not exist"
Jul 28 16:34:41.654249 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.654193    2555 scope.go:117] "RemoveContainer" containerID="ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"
Jul 28 16:34:41.654424 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:41.654408    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65\": container with ID starting with ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65 not found: ID does not exist" containerID="ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"
Jul 28 16:34:41.654487 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.654426    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65"} err="failed to get container status \"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65\": rpc error: code = NotFound desc = could not find container \"ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65\": container with ID starting with ae61dd42fcd8078a060356ced4d7c271bd93ff9e76d0639bd765e1dd69e74f65 not found: ID does not exist"
Jul 28 16:34:41.654487 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.654437    2555 scope.go:117] "RemoveContainer" containerID="8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"
Jul 28 16:34:41.654622 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:34:41.654605    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d\": container with ID starting with 8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d not found: ID does not exist" containerID="8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"
Jul 28 16:34:41.654680 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.654623    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d"} err="failed to get container status \"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d\": rpc error: code = NotFound desc = could not find container \"8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d\": container with ID starting with 8d5805fd5f4c1c20d97256e21a591e0179ed25de0c817c959c0511117464771d not found: ID does not exist"
Jul 28 16:34:41.706757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.706728    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-cabundle-cert\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:41.706757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.706752    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdb8\" (UniqueName: \"kubernetes.io/projected/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-api-access-rvdb8\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:41.706757 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.706763    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8a1a80a-9dcb-4596-be10-bf900623d462-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:41.706950 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.706771    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8a1a80a-9dcb-4596-be10-bf900623d462-kserve-provision-location\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:41.706950 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.706780    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8a1a80a-9dcb-4596-be10-bf900623d462-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:34:41.824116 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:41.824077    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" path="/var/lib/kubelet/pods/e8a1a80a-9dcb-4596-be10-bf900623d462/volumes"
Jul 28 16:34:43.573765 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:43.573730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:44.621268 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:44.621242    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:34:44.621693 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:44.621635    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:34:44.622380 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:44.622361    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:34:48.573329 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:48.573246    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:48.573714 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:48.573362    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:34:53.573977 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:53.573933    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:54.621686 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:54.621644    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:34:58.573538 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:34:58.573499    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:03.573847 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:03.573799    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:04.621643 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:04.621605    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:35:06.970821 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:06.970792    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:35:07.104853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.104770    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls\") pod \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") "
Jul 28 16:35:07.104853 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.104828    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle\") pod \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\" (UID: \"0bfe2578-34f0-4b86-8d74-5045ab27a60f\") "
Jul 28 16:35:07.105227 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.105206    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0bfe2578-34f0-4b86-8d74-5045ab27a60f" (UID: "0bfe2578-34f0-4b86-8d74-5045ab27a60f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:07.107013 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.106981    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bfe2578-34f0-4b86-8d74-5045ab27a60f" (UID: "0bfe2578-34f0-4b86-8d74-5045ab27a60f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:35:07.205797 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.205761    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bfe2578-34f0-4b86-8d74-5045ab27a60f-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:35:07.205797 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.205791    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bfe2578-34f0-4b86-8d74-5045ab27a60f-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:35:07.724091 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.724061    2555 generic.go:356] "Generic (PLEG): container finished" podID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerID="d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161" exitCode=0
Jul 28 16:35:07.724272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.724107    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"
Jul 28 16:35:07.724272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.724152    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" event={"ID":"0bfe2578-34f0-4b86-8d74-5045ab27a60f","Type":"ContainerDied","Data":"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"}
Jul 28 16:35:07.724272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.724196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh" event={"ID":"0bfe2578-34f0-4b86-8d74-5045ab27a60f","Type":"ContainerDied","Data":"07743da5c7b46e9de72a8bb54add96450988526a5b80ec66741b0940042a237a"}
Jul 28 16:35:07.724272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.724219    2555 scope.go:117] "RemoveContainer" containerID="d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"
Jul 28 16:35:07.732493 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.732468    2555 scope.go:117] "RemoveContainer" containerID="d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"
Jul 28 16:35:07.732742 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:35:07.732725    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161\": container with ID starting with d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161 not found: ID does not exist" containerID="d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"
Jul 28 16:35:07.732800 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.732749    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161"} err="failed to get container status \"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161\": rpc error: code = NotFound desc = could not find container \"d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161\": container with ID starting with d983296e036acf17d9d98d2fb25e659b0e677b58f4a956ef62dd0d15b241b161 not found: ID does not exist"
Jul 28 16:35:07.742368 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.742348    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:35:07.744761 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.744743    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c5b4df566-hptsh"]
Jul 28 16:35:07.823275 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:07.823250    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" path="/var/lib/kubelet/pods/0bfe2578-34f0-4b86-8d74-5045ab27a60f/volumes"
Jul 28 16:35:14.622545 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:14.622511    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:35:24.623077 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:24.623050    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:35:37.027406 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027371    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027652    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027663    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027675    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kube-rbac-proxy"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027680    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kube-rbac-proxy"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027692    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="storage-initializer"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027697    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="storage-initializer"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027708    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer"
Jul 28 16:35:37.027739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027713    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer"
Jul 28 16:35:37.027967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027758    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bfe2578-34f0-4b86-8d74-5045ab27a60f" containerName="model-chainer"
Jul 28 16:35:37.027967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027764    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kserve-container"
Jul 28 16:35:37.027967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.027769    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a1a80a-9dcb-4596-be10-bf900623d462" containerName="kube-rbac-proxy"
Jul 28 16:35:37.031854 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.031836    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.033929 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.033906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2b606-kube-rbac-proxy-sar-config\""
Jul 28 16:35:37.034059 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.033961    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2b606-serving-cert\""
Jul 28 16:35:37.039889 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.039857    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:35:37.102538 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.102509    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.102538 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.102538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.203143 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.203113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.203278 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.203149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.203278 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:35:37.203266    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-2b606-serving-cert: secret "sequence-graph-2b606-serving-cert" not found
Jul 28 16:35:37.203393 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:35:37.203343    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls podName:c22927ab-2190-4388-b85f-4b8214bad77a nodeName:}" failed. No retries permitted until 2026-07-28 16:35:37.703323778 +0000 UTC m=+852.435953436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls") pod "sequence-graph-2b606-679788d849-5mxsm" (UID: "c22927ab-2190-4388-b85f-4b8214bad77a") : secret "sequence-graph-2b606-serving-cert" not found
Jul 28 16:35:37.203695 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.203674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.706356 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.706315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.708686 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.708668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") pod \"sequence-graph-2b606-679788d849-5mxsm\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") " pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:37.942174 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:37.942146    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:38.061391 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:38.061369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:35:38.063721 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:35:38.063696    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22927ab_2190_4388_b85f_4b8214bad77a.slice/crio-94f6bacaeb3cba056d53ad0e22046b50324611f284e962e1368239222cf7807c WatchSource:0}: Error finding container 94f6bacaeb3cba056d53ad0e22046b50324611f284e962e1368239222cf7807c: Status 404 returned error can't find the container with id 94f6bacaeb3cba056d53ad0e22046b50324611f284e962e1368239222cf7807c
Jul 28 16:35:38.836693 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:38.836657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" event={"ID":"c22927ab-2190-4388-b85f-4b8214bad77a","Type":"ContainerStarted","Data":"66691bc801a5086997eace1fa95aac2e31969ff1bde93bfaca4b80367c6a36c8"}
Jul 28 16:35:38.836880 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:38.836713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" event={"ID":"c22927ab-2190-4388-b85f-4b8214bad77a","Type":"ContainerStarted","Data":"94f6bacaeb3cba056d53ad0e22046b50324611f284e962e1368239222cf7807c"}
Jul 28 16:35:38.836880 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:38.836770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:35:38.854241 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:38.854200    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podStartSLOduration=1.854187246 podStartE2EDuration="1.854187246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:35:38.853276988 +0000 UTC m=+853.585906664" watchObservedRunningTime="2026-07-28 16:35:38.854187246 +0000 UTC m=+853.586816923"
Jul 28 16:35:44.845276 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:35:44.845248    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:36:25.657447 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:36:25.657417    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:36:25.660337 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:36:25.660304    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:41:25.677994 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:41:25.677914    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:41:25.682543 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:41:25.682524    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:42:51.679677 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.679649    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:42:51.682135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.679871    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" containerID="cri-o://1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e" gracePeriod=30
Jul 28 16:42:51.908452 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.908424    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:42:51.908672 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.908649    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kube-rbac-proxy" containerID="cri-o://4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a" gracePeriod=30
Jul 28 16:42:51.908752 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.908637    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container" containerID="cri-o://a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11" gracePeriod=30
Jul 28 16:42:51.930356 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.930303    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:42:51.933349 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.933333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:51.935624 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.935606    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-68c6a-predictor-serving-cert\""
Jul 28 16:42:51.935863 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.935849    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-68c6a-kube-rbac-proxy-sar-config\""
Jul 28 16:42:51.942621 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.942600    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:42:51.994678 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.994646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:51.994802 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.994702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:51.994802 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:51.994738    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q78t\" (UniqueName: \"kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.095842 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.095812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.095959 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.095846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q78t\" (UniqueName: \"kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.096034 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.095986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.096170 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:42:52.096154    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-serving-cert: secret "error-404-isvc-68c6a-predictor-serving-cert" not found
Jul 28 16:42:52.096248 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:42:52.096238    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls podName:53558dfa-f244-4567-a33e-02df75873ba1 nodeName:}" failed. No retries permitted until 2026-07-28 16:42:52.596217338 +0000 UTC m=+1287.328846997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls") pod "error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" (UID: "53558dfa-f244-4567-a33e-02df75873ba1") : secret "error-404-isvc-68c6a-predictor-serving-cert" not found
Jul 28 16:42:52.096548 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.096532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.105876 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.105844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q78t\" (UniqueName: \"kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.334288 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.334259    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerID="4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a" exitCode=2
Jul 28 16:42:52.334452 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.334342    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerDied","Data":"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"}
Jul 28 16:42:52.600630 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.600547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.602916 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.602899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") pod \"error-404-isvc-68c6a-predictor-846757f6bb-2ss5d\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") " pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.843162 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.843125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:52.964481 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:52.964449    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:42:52.967184 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:42:52.967152    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53558dfa_f244_4567_a33e_02df75873ba1.slice/crio-178a967f0a48d08ec5e9471351ef5535c63ad6afffca3f2a0321300cb23b3c91 WatchSource:0}: Error finding container 178a967f0a48d08ec5e9471351ef5535c63ad6afffca3f2a0321300cb23b3c91: Status 404 returned error can't find the container with id 178a967f0a48d08ec5e9471351ef5535c63ad6afffca3f2a0321300cb23b3c91
Jul 28 16:42:53.339730 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.339697    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerStarted","Data":"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"}
Jul 28 16:42:53.339891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.339759    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerStarted","Data":"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"}
Jul 28 16:42:53.339891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.339784    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:53.339891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.339810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerStarted","Data":"178a967f0a48d08ec5e9471351ef5535c63ad6afffca3f2a0321300cb23b3c91"}
Jul 28 16:42:53.340040 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.339913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:53.341185 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.341164    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:42:53.356750 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:53.356707    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podStartSLOduration=2.356692969 podStartE2EDuration="2.356692969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:42:53.355075973 +0000 UTC m=+1288.087705651" watchObservedRunningTime="2026-07-28 16:42:53.356692969 +0000 UTC m=+1288.089322646"
Jul 28 16:42:54.343863 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.343830    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:42:54.620662 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.620569    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:42:54.853385 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.853363    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:42:54.918188 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.918123    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config\") pod \"6a14f67c-97b7-4857-a8f1-61bae732ad19\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") "
Jul 28 16:42:54.918339 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.918203    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls\") pod \"6a14f67c-97b7-4857-a8f1-61bae732ad19\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") "
Jul 28 16:42:54.918339 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.918252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh765\" (UniqueName: \"kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765\") pod \"6a14f67c-97b7-4857-a8f1-61bae732ad19\" (UID: \"6a14f67c-97b7-4857-a8f1-61bae732ad19\") "
Jul 28 16:42:54.918464 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.918440    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a14f67c-97b7-4857-a8f1-61bae732ad19" (UID: "6a14f67c-97b7-4857-a8f1-61bae732ad19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:54.920275 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.920252    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765" (OuterVolumeSpecName: "kube-api-access-gh765") pod "6a14f67c-97b7-4857-a8f1-61bae732ad19" (UID: "6a14f67c-97b7-4857-a8f1-61bae732ad19"). InnerVolumeSpecName "kube-api-access-gh765". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:42:54.920385 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:54.920366    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a14f67c-97b7-4857-a8f1-61bae732ad19" (UID: "6a14f67c-97b7-4857-a8f1-61bae732ad19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:42:55.018839 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.018803    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a14f67c-97b7-4857-a8f1-61bae732ad19-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:42:55.018839 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.018835    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh765\" (UniqueName: \"kubernetes.io/projected/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-api-access-gh765\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:42:55.018839 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.018845    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a14f67c-97b7-4857-a8f1-61bae732ad19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:42:55.348057 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.348027    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerID="a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11" exitCode=0
Jul 28 16:42:55.348462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.348088    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"
Jul 28 16:42:55.348462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.348106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerDied","Data":"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"}
Jul 28 16:42:55.348462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.348193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv" event={"ID":"6a14f67c-97b7-4857-a8f1-61bae732ad19","Type":"ContainerDied","Data":"862d43e1edb49b213c0bae2e8b8c554d4cf87b6a14e0813f490f8783e1663c09"}
Jul 28 16:42:55.348462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.348220    2555 scope.go:117] "RemoveContainer" containerID="4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"
Jul 28 16:42:55.356412 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.356392    2555 scope.go:117] "RemoveContainer" containerID="a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"
Jul 28 16:42:55.363317 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.363301    2555 scope.go:117] "RemoveContainer" containerID="4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"
Jul 28 16:42:55.363566 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:42:55.363547    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a\": container with ID starting with 4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a not found: ID does not exist" containerID="4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"
Jul 28 16:42:55.363602 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.363575    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a"} err="failed to get container status \"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a\": rpc error: code = NotFound desc = could not find container \"4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a\": container with ID starting with 4209129782bf5266614045880bdb8ab5680f32645ce24ec4c19cb519a5590e5a not found: ID does not exist"
Jul 28 16:42:55.363602 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.363593    2555 scope.go:117] "RemoveContainer" containerID="a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"
Jul 28 16:42:55.363790 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:42:55.363777    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11\": container with ID starting with a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11 not found: ID does not exist" containerID="a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"
Jul 28 16:42:55.363825 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.363794    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11"} err="failed to get container status \"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11\": rpc error: code = NotFound desc = could not find container \"a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11\": container with ID starting with a08bbe6f341a501c719b9b7e9e2c2b3ee760ea12f13a6908f34a001302491e11 not found: ID does not exist"
Jul 28 16:42:55.370670 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.370648    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:42:55.372403 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.372385    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ba9d-predictor-59b97b6f68-jvqvv"]
Jul 28 16:42:55.823106 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:55.823069    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" path="/var/lib/kubelet/pods/6a14f67c-97b7-4857-a8f1-61bae732ad19/volumes"
Jul 28 16:42:59.349291 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:59.349266    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:42:59.349697 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:59.349671    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:42:59.620954 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:42:59.620875    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:04.620892 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:04.620848    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:04.621363 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:04.620972    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:43:09.349908 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:09.349858    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:43:09.620893 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:09.620811    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:14.620782 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:14.620745    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:19.350074 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:19.350036    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:43:19.621264 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:19.621169    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:22.333380 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.333351    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:43:22.417996 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.417963    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle\") pod \"b1e6379f-d775-4c57-bd47-d9e00c96f420\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") "
Jul 28 16:43:22.418177 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.418061    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") pod \"b1e6379f-d775-4c57-bd47-d9e00c96f420\" (UID: \"b1e6379f-d775-4c57-bd47-d9e00c96f420\") "
Jul 28 16:43:22.418314 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.418291    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b1e6379f-d775-4c57-bd47-d9e00c96f420" (UID: "b1e6379f-d775-4c57-bd47-d9e00c96f420"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:43:22.420183 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.420153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1e6379f-d775-4c57-bd47-d9e00c96f420" (UID: "b1e6379f-d775-4c57-bd47-d9e00c96f420"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:43:22.450736 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.450669    2555 generic.go:356] "Generic (PLEG): container finished" podID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerID="1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e" exitCode=0
Jul 28 16:43:22.450841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.450743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" event={"ID":"b1e6379f-d775-4c57-bd47-d9e00c96f420","Type":"ContainerDied","Data":"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"}
Jul 28 16:43:22.450841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.450766    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"
Jul 28 16:43:22.450841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.450817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk" event={"ID":"b1e6379f-d775-4c57-bd47-d9e00c96f420","Type":"ContainerDied","Data":"4d87db0102a8c819a9f5a2fcebab2e3c7764129156c8a8487ea72b8aef24bfb8"}
Jul 28 16:43:22.450841 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.450836    2555 scope.go:117] "RemoveContainer" containerID="1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"
Jul 28 16:43:22.458940 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.458925    2555 scope.go:117] "RemoveContainer" containerID="1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"
Jul 28 16:43:22.459209 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:43:22.459190    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e\": container with ID starting with 1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e not found: ID does not exist" containerID="1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"
Jul 28 16:43:22.459265 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.459217    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e"} err="failed to get container status \"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e\": rpc error: code = NotFound desc = could not find container \"1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e\": container with ID starting with 1c56505055b61fac56232c9afc8c7622dd189c269e53227c5e6abf4ae4d8dc5e not found: ID does not exist"
Jul 28 16:43:22.471320 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.471299    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:43:22.474614 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.474595    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7ba9d-5757fd8cc-68rtk"]
Jul 28 16:43:22.519272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.519250    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e6379f-d775-4c57-bd47-d9e00c96f420-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:43:22.519272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:22.519271    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e6379f-d775-4c57-bd47-d9e00c96f420-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:43:23.823504 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:23.823468    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" path="/var/lib/kubelet/pods/b1e6379f-d775-4c57-bd47-d9e00c96f420/volumes"
Jul 28 16:43:29.350093 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:29.350050    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:43:39.351100 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:39.351073    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:43:51.663169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.663092    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:43:51.663526 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.663323    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" containerID="cri-o://66691bc801a5086997eace1fa95aac2e31969ff1bde93bfaca4b80367c6a36c8" gracePeriod=30
Jul 28 16:43:51.951250 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.951175    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:43:51.951427 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.951388    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" containerID="cri-o://7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9" gracePeriod=30
Jul 28 16:43:51.951484 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.951418    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kube-rbac-proxy" containerID="cri-o://93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152" gracePeriod=30
Jul 28 16:43:51.995898 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.995874    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:43:51.996282 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996264    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container"
Jul 28 16:43:51.996359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996291    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container"
Jul 28 16:43:51.996359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996346    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kube-rbac-proxy"
Jul 28 16:43:51.996359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996353    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kube-rbac-proxy"
Jul 28 16:43:51.996359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996360    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d"
Jul 28 16:43:51.996359 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996365    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d"
Jul 28 16:43:51.996617 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kserve-container"
Jul 28 16:43:51.996617 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996441    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e6379f-d775-4c57-bd47-d9e00c96f420" containerName="switch-graph-7ba9d"
Jul 28 16:43:51.996617 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.996454    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a14f67c-97b7-4857-a8f1-61bae732ad19" containerName="kube-rbac-proxy"
Jul 28 16:43:51.999388 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:51.999372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.004423 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.004405    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-68c6a-serving-cert\""
Jul 28 16:43:52.014984 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.014954    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-68c6a-kube-rbac-proxy-sar-config\""
Jul 28 16:43:52.021286 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.021265    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:43:52.033966 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.033945    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:43:52.037221 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.037207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.040548 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.040532    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92dc3-predictor-serving-cert\""
Jul 28 16:43:52.041602 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.041585    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92dc3-kube-rbac-proxy-sar-config\""
Jul 28 16:43:52.054053 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.054033    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:43:52.131293 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.131265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.131426 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.131297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.231941 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.231871    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.231941 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.231909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.231941 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.231929    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.232205 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.231952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.232205 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.232040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mnn\" (UniqueName: \"kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.232478 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.232461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.234475 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.234453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls\") pod \"ensemble-graph-68c6a-6655b859d7-hlhjf\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") " pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.308272 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.308253    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.333340 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.333317    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.333452 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.333350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.333452 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.333375    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mnn\" (UniqueName: \"kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.333975 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.333952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.335914 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.335892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.342711 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.342693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mnn\" (UniqueName: \"kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn\") pod \"error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") " pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.346540 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.346523    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:52.450920 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.450856    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:43:52.453270 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:43:52.453219    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee6ca21_44cc_4acd_bcba_4de270914ddd.slice/crio-0b96ee071890a38d4235abed4e4853b06b696c82d86c007bf2fe3bd89ecd111f WatchSource:0}: Error finding container 0b96ee071890a38d4235abed4e4853b06b696c82d86c007bf2fe3bd89ecd111f: Status 404 returned error can't find the container with id 0b96ee071890a38d4235abed4e4853b06b696c82d86c007bf2fe3bd89ecd111f
Jul 28 16:43:52.491454 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.491428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:43:52.493855 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:43:52.493830    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c4522c_7798_426e_a390_5163faea5e7c.slice/crio-6926d7327f5fc048ba4a907453ca8922d11f2eb79a7608105f8b5b38392917a6 WatchSource:0}: Error finding container 6926d7327f5fc048ba4a907453ca8922d11f2eb79a7608105f8b5b38392917a6: Status 404 returned error can't find the container with id 6926d7327f5fc048ba4a907453ca8922d11f2eb79a7608105f8b5b38392917a6
Jul 28 16:43:52.561940 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.561915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerStarted","Data":"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"}
Jul 28 16:43:52.562066 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.561973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerStarted","Data":"6926d7327f5fc048ba4a907453ca8922d11f2eb79a7608105f8b5b38392917a6"}
Jul 28 16:43:52.563560 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.563539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" event={"ID":"fee6ca21-44cc-4acd-bcba-4de270914ddd","Type":"ContainerStarted","Data":"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"}
Jul 28 16:43:52.563656 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.563578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" event={"ID":"fee6ca21-44cc-4acd-bcba-4de270914ddd","Type":"ContainerStarted","Data":"0b96ee071890a38d4235abed4e4853b06b696c82d86c007bf2fe3bd89ecd111f"}
Jul 28 16:43:52.563656 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.563611    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:52.565387 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.565366    2555 generic.go:356] "Generic (PLEG): container finished" podID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerID="93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152" exitCode=2
Jul 28 16:43:52.565471 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.565418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerDied","Data":"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"}
Jul 28 16:43:52.581094 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:52.581059    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podStartSLOduration=1.5810463970000002 podStartE2EDuration="1.581046397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:43:52.579962516 +0000 UTC m=+1347.312592192" watchObservedRunningTime="2026-07-28 16:43:52.581046397 +0000 UTC m=+1347.313676074"
Jul 28 16:43:53.571489 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:53.571454    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerStarted","Data":"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"}
Jul 28 16:43:53.590267 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:53.590218    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podStartSLOduration=1.590200294 podStartE2EDuration="1.590200294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:43:53.589778858 +0000 UTC m=+1348.322408536" watchObservedRunningTime="2026-07-28 16:43:53.590200294 +0000 UTC m=+1348.322829970"
Jul 28 16:43:54.576488 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.576463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:54.576919 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.576515    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:43:54.577750 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.577726    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:43:54.617640 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.617610    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 16:43:54.621867 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.621847    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 16:43:54.793759 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.793737    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:43:54.843744 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.843666    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:43:54.949422 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.949394    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xtng\" (UniqueName: \"kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng\") pod \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") "
Jul 28 16:43:54.949543 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.949446    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config\") pod \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") "
Jul 28 16:43:54.949543 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.949480    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") pod \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\" (UID: \"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0\") "
Jul 28 16:43:54.949837 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.949809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" (UID: "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:43:54.951634 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.951614    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" (UID: "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:43:54.951935 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:54.951920    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng" (OuterVolumeSpecName: "kube-api-access-5xtng") pod "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" (UID: "6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0"). InnerVolumeSpecName "kube-api-access-5xtng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:43:55.050403 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.050380    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:43:55.050403 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.050400    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:43:55.050528 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.050411    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtng\" (UniqueName: \"kubernetes.io/projected/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0-kube-api-access-5xtng\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:43:55.581472 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.581441    2555 generic.go:356] "Generic (PLEG): container finished" podID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerID="7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9" exitCode=0
Jul 28 16:43:55.581974 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.581499    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"
Jul 28 16:43:55.581974 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.581529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerDied","Data":"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"}
Jul 28 16:43:55.581974 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.581588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86" event={"ID":"6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0","Type":"ContainerDied","Data":"61c6f427db7a19cc8480c247bb3efa217b831878b53db2110ab2cda8ce0bc2e9"}
Jul 28 16:43:55.581974 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.581613    2555 scope.go:117] "RemoveContainer" containerID="93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"
Jul 28 16:43:55.582219 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.582173    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:43:55.590061 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.590046    2555 scope.go:117] "RemoveContainer" containerID="7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"
Jul 28 16:43:55.597082 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.597066    2555 scope.go:117] "RemoveContainer" containerID="93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"
Jul 28 16:43:55.597413 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:43:55.597393    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152\": container with ID starting with 93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152 not found: ID does not exist" containerID="93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"
Jul 28 16:43:55.597463 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.597422    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152"} err="failed to get container status \"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152\": rpc error: code = NotFound desc = could not find container \"93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152\": container with ID starting with 93c704c15b80fdefb15080041ba9b4ecd4eba6626207002745a247c31b26d152 not found: ID does not exist"
Jul 28 16:43:55.597463 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.597436    2555 scope.go:117] "RemoveContainer" containerID="7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"
Jul 28 16:43:55.597647 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:43:55.597633    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9\": container with ID starting with 7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9 not found: ID does not exist" containerID="7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"
Jul 28 16:43:55.597681 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.597652    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9"} err="failed to get container status \"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9\": rpc error: code = NotFound desc = could not find container \"7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9\": container with ID starting with 7cb90791c116eaa84e44c47a0fe62dfdef5b038d71d049d9d5c83a4dafd40bb9 not found: ID does not exist"
Jul 28 16:43:55.600755 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.600737    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:43:55.604413 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.604396    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2b606-predictor-76cf8477c8-wnb86"]
Jul 28 16:43:55.823347 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:55.823320    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" path="/var/lib/kubelet/pods/6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0/volumes"
Jul 28 16:43:58.576973 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:58.576948    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:43:59.843508 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:43:59.843471    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:00.586866 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:00.586839    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:44:00.587350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:00.587324    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:44:01.969797 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:01.969724    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:44:01.970236 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:01.969968    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" containerID="cri-o://b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a" gracePeriod=30
Jul 28 16:44:02.182742 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.182711    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:44:02.182990 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.182952    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container" containerID="cri-o://f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34" gracePeriod=30
Jul 28 16:44:02.183147 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.182996    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kube-rbac-proxy" containerID="cri-o://774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507" gracePeriod=30
Jul 28 16:44:02.214919 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.214891    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:44:02.215191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215179    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container"
Jul 28 16:44:02.215237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215192    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container"
Jul 28 16:44:02.215237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215205    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kube-rbac-proxy"
Jul 28 16:44:02.215237 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215211    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kube-rbac-proxy"
Jul 28 16:44:02.215324 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215276    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kube-rbac-proxy"
Jul 28 16:44:02.215324 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.215285    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec7b3f7-6eae-42e8-9e8c-364b14e5eed0" containerName="kserve-container"
Jul 28 16:44:02.219389 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.219372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.221710 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.221669    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3541-predictor-serving-cert\""
Jul 28 16:44:02.221710 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.221702    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3541-kube-rbac-proxy-sar-config\""
Jul 28 16:44:02.226971 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.226951    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:44:02.303335 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.303312    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.303461 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.303348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8hd\" (UniqueName: \"kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.303461 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.303381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.404191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.404153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.404191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.404203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8hd\" (UniqueName: \"kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.404385 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.404234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.404839 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.404821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.406651 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.406634    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.411639 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.411619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8hd\" (UniqueName: \"kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd\") pod \"error-404-isvc-c3541-predictor-67c685758b-jbrfq\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") " pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.529983 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.529960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:02.607900 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.607872    2555 generic.go:356] "Generic (PLEG): container finished" podID="53558dfa-f244-4567-a33e-02df75873ba1" containerID="774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507" exitCode=2
Jul 28 16:44:02.608057 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.607927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerDied","Data":"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"}
Jul 28 16:44:02.657599 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:02.657575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:44:02.660154 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:44:02.660128    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2edb5f1_1799_4f12_af81_4659496e8837.slice/crio-3b0343492c72debb776352e7e4efa8a557eedb23a6f6ab0e977afe337c655207 WatchSource:0}: Error finding container 3b0343492c72debb776352e7e4efa8a557eedb23a6f6ab0e977afe337c655207: Status 404 returned error can't find the container with id 3b0343492c72debb776352e7e4efa8a557eedb23a6f6ab0e977afe337c655207
Jul 28 16:44:03.575861 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.575820    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:03.613558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.613523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerStarted","Data":"8982ff89ee0a1e050645dda3ef88189679452b81a1e55e8a719c6897a5876833"}
Jul 28 16:44:03.613691 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.613589    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerStarted","Data":"b8aaf862aad6446fbcba17b549e0a7f176db782e6eaa039e155fbfae80935258"}
Jul 28 16:44:03.613691 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.613621    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerStarted","Data":"3b0343492c72debb776352e7e4efa8a557eedb23a6f6ab0e977afe337c655207"}
Jul 28 16:44:03.613780 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.613752    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:03.613814 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.613801    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:03.614971 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.614946    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:03.634103 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:03.634068    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podStartSLOduration=1.634055628 podStartE2EDuration="1.634055628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:44:03.632841775 +0000 UTC m=+1358.365471475" watchObservedRunningTime="2026-07-28 16:44:03.634055628 +0000 UTC m=+1358.366685303"
Jul 28 16:44:04.344640 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:04.344604    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 16:44:04.617692 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:04.617606    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:04.843531 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:04.843494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:04.843695 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:04.843595    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:44:05.622350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.622318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:44:05.622350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.622343    2555 generic.go:356] "Generic (PLEG): container finished" podID="53558dfa-f244-4567-a33e-02df75873ba1" containerID="f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34" exitCode=0
Jul 28 16:44:05.622650 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.622417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerDied","Data":"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"}
Jul 28 16:44:05.622650 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.622496    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d" event={"ID":"53558dfa-f244-4567-a33e-02df75873ba1","Type":"ContainerDied","Data":"178a967f0a48d08ec5e9471351ef5535c63ad6afffca3f2a0321300cb23b3c91"}
Jul 28 16:44:05.622650 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.622524    2555 scope.go:117] "RemoveContainer" containerID="774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"
Jul 28 16:44:05.628095 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.628077    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q78t\" (UniqueName: \"kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t\") pod \"53558dfa-f244-4567-a33e-02df75873ba1\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") "
Jul 28 16:44:05.628191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.628121    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config\") pod \"53558dfa-f244-4567-a33e-02df75873ba1\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") "
Jul 28 16:44:05.628191 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.628162    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") pod \"53558dfa-f244-4567-a33e-02df75873ba1\" (UID: \"53558dfa-f244-4567-a33e-02df75873ba1\") "
Jul 28 16:44:05.628460 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.628441    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53558dfa-f244-4567-a33e-02df75873ba1" (UID: "53558dfa-f244-4567-a33e-02df75873ba1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:05.630121 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.630106    2555 scope.go:117] "RemoveContainer" containerID="f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"
Jul 28 16:44:05.630206 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.630187    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53558dfa-f244-4567-a33e-02df75873ba1" (UID: "53558dfa-f244-4567-a33e-02df75873ba1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:44:05.630457 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.630437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t" (OuterVolumeSpecName: "kube-api-access-7q78t") pod "53558dfa-f244-4567-a33e-02df75873ba1" (UID: "53558dfa-f244-4567-a33e-02df75873ba1"). InnerVolumeSpecName "kube-api-access-7q78t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:44:05.643385 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.643363    2555 scope.go:117] "RemoveContainer" containerID="774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"
Jul 28 16:44:05.643657 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:44:05.643623    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507\": container with ID starting with 774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507 not found: ID does not exist" containerID="774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"
Jul 28 16:44:05.643750 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.643658    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507"} err="failed to get container status \"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507\": rpc error: code = NotFound desc = could not find container \"774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507\": container with ID starting with 774057e62ced0c119d8884f85c91061bcbc13e4569a71e2546323db0b1fc3507 not found: ID does not exist"
Jul 28 16:44:05.643750 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.643681    2555 scope.go:117] "RemoveContainer" containerID="f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"
Jul 28 16:44:05.643951 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:44:05.643935    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34\": container with ID starting with f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34 not found: ID does not exist" containerID="f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"
Jul 28 16:44:05.643994 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.643957    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34"} err="failed to get container status \"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34\": rpc error: code = NotFound desc = could not find container \"f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34\": container with ID starting with f6cfc2121ef7d49cfe83753ce109ca5eae47f237687c091e49f4123d554ccd34 not found: ID does not exist"
Jul 28 16:44:05.728824 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.728761    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q78t\" (UniqueName: \"kubernetes.io/projected/53558dfa-f244-4567-a33e-02df75873ba1-kube-api-access-7q78t\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:05.728824 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.728782    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53558dfa-f244-4567-a33e-02df75873ba1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:05.728824 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:05.728792    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53558dfa-f244-4567-a33e-02df75873ba1-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:06.627915 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:06.627889    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"
Jul 28 16:44:06.645756 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:06.645732    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:44:06.649842 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:06.649817    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68c6a-predictor-846757f6bb-2ss5d"]
Jul 28 16:44:07.822915 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:07.822883    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53558dfa-f244-4567-a33e-02df75873ba1" path="/var/lib/kubelet/pods/53558dfa-f244-4567-a33e-02df75873ba1/volumes"
Jul 28 16:44:08.575865 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:08.575832    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:09.622488 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:09.622463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:09.622896 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:09.622874    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:09.843930 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:09.843897    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:10.587714 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:10.587678    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:44:13.575487 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:13.575455    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:13.575829 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:13.575551    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:44:14.843244 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:14.843212    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:18.576297 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:18.576264    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:19.623641 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:19.623604    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:19.844115 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:19.844077    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:20.587573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:20.587536    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:44:21.688894 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.688869    2555 generic.go:356] "Generic (PLEG): container finished" podID="c22927ab-2190-4388-b85f-4b8214bad77a" containerID="66691bc801a5086997eace1fa95aac2e31969ff1bde93bfaca4b80367c6a36c8" exitCode=0
Jul 28 16:44:21.689195 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.688942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" event={"ID":"c22927ab-2190-4388-b85f-4b8214bad77a","Type":"ContainerDied","Data":"66691bc801a5086997eace1fa95aac2e31969ff1bde93bfaca4b80367c6a36c8"}
Jul 28 16:44:21.805202 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.805181    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:44:21.841596 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.841573    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") pod \"c22927ab-2190-4388-b85f-4b8214bad77a\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") "
Jul 28 16:44:21.841735 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.841611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle\") pod \"c22927ab-2190-4388-b85f-4b8214bad77a\" (UID: \"c22927ab-2190-4388-b85f-4b8214bad77a\") "
Jul 28 16:44:21.841962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.841940    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c22927ab-2190-4388-b85f-4b8214bad77a" (UID: "c22927ab-2190-4388-b85f-4b8214bad77a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:21.843591 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.843567    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c22927ab-2190-4388-b85f-4b8214bad77a" (UID: "c22927ab-2190-4388-b85f-4b8214bad77a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:44:21.942785 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.942714    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22927ab-2190-4388-b85f-4b8214bad77a-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:21.942785 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:21.942740    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22927ab-2190-4388-b85f-4b8214bad77a-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:22.693761 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:22.693725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm" event={"ID":"c22927ab-2190-4388-b85f-4b8214bad77a","Type":"ContainerDied","Data":"94f6bacaeb3cba056d53ad0e22046b50324611f284e962e1368239222cf7807c"}
Jul 28 16:44:22.693761 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:22.693751    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"
Jul 28 16:44:22.694273 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:22.693819    2555 scope.go:117] "RemoveContainer" containerID="66691bc801a5086997eace1fa95aac2e31969ff1bde93bfaca4b80367c6a36c8"
Jul 28 16:44:22.720969 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:22.720948    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:44:22.728311 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:22.728290    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2b606-679788d849-5mxsm"]
Jul 28 16:44:23.575730 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:23.575696    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:23.823131 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:23.823099    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" path="/var/lib/kubelet/pods/c22927ab-2190-4388-b85f-4b8214bad77a/volumes"
Jul 28 16:44:28.575739 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:28.575696    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:44:29.623530 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:29.623492    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:30.587518 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:30.587480    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:44:32.107112 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.107089    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:44:32.217580 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.217555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls\") pod \"fee6ca21-44cc-4acd-bcba-4de270914ddd\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") "
Jul 28 16:44:32.217744 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.217607    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle\") pod \"fee6ca21-44cc-4acd-bcba-4de270914ddd\" (UID: \"fee6ca21-44cc-4acd-bcba-4de270914ddd\") "
Jul 28 16:44:32.217963 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.217937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fee6ca21-44cc-4acd-bcba-4de270914ddd" (UID: "fee6ca21-44cc-4acd-bcba-4de270914ddd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:32.219607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.219584    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fee6ca21-44cc-4acd-bcba-4de270914ddd" (UID: "fee6ca21-44cc-4acd-bcba-4de270914ddd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:44:32.318722 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.318691    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee6ca21-44cc-4acd-bcba-4de270914ddd-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:32.318722 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.318723    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee6ca21-44cc-4acd-bcba-4de270914ddd-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:44:32.731239 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.731159    2555 generic.go:356] "Generic (PLEG): container finished" podID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerID="b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a" exitCode=0
Jul 28 16:44:32.731239 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.731215    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"
Jul 28 16:44:32.731239 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.731233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" event={"ID":"fee6ca21-44cc-4acd-bcba-4de270914ddd","Type":"ContainerDied","Data":"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"}
Jul 28 16:44:32.731464 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.731306    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf" event={"ID":"fee6ca21-44cc-4acd-bcba-4de270914ddd","Type":"ContainerDied","Data":"0b96ee071890a38d4235abed4e4853b06b696c82d86c007bf2fe3bd89ecd111f"}
Jul 28 16:44:32.731464 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.731330    2555 scope.go:117] "RemoveContainer" containerID="b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"
Jul 28 16:44:32.739334 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.739317    2555 scope.go:117] "RemoveContainer" containerID="b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"
Jul 28 16:44:32.739554 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:44:32.739537    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a\": container with ID starting with b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a not found: ID does not exist" containerID="b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"
Jul 28 16:44:32.739597 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.739563    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a"} err="failed to get container status \"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a\": rpc error: code = NotFound desc = could not find container \"b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a\": container with ID starting with b67872be000a19d802792487836c9f43f0116d7d205af36f0db3f1337e9d1c2a not found: ID does not exist"
Jul 28 16:44:32.751256 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.751234    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:44:32.756544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:32.756527    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-68c6a-6655b859d7-hlhjf"]
Jul 28 16:44:33.823129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:33.823094    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" path="/var/lib/kubelet/pods/fee6ca21-44cc-4acd-bcba-4de270914ddd/volumes"
Jul 28 16:44:39.623042 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:39.622988    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:44:40.588778 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:40.588748    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:44:49.624194 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:49.624165    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:44:51.830022 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.829982    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830267    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830278    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830290    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kube-rbac-proxy"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kube-rbac-proxy"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830304    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830309    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830316    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a"
Jul 28 16:44:51.830361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830320    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a"
Jul 28 16:44:51.830592 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830368    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kserve-container"
Jul 28 16:44:51.830592 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830376    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22927ab-2190-4388-b85f-4b8214bad77a" containerName="sequence-graph-2b606"
Jul 28 16:44:51.830592 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830382    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53558dfa-f244-4567-a33e-02df75873ba1" containerName="kube-rbac-proxy"
Jul 28 16:44:51.830592 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.830388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee6ca21-44cc-4acd-bcba-4de270914ddd" containerName="ensemble-graph-68c6a"
Jul 28 16:44:51.834491 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.834471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:51.836708 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.836689    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-92dc3-kube-rbac-proxy-sar-config\""
Jul 28 16:44:51.836773 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.836730    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-92dc3-serving-cert\""
Jul 28 16:44:51.842490 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.842470    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:44:51.952888 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.952863    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:51.953082 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:51.952895    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.053798 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.053774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.053903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.053803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.053903 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:44:52.053885    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-92dc3-serving-cert: secret "sequence-graph-92dc3-serving-cert" not found
Jul 28 16:44:52.053973 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:44:52.053937    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls podName:46788321-15b6-4aa8-ad9e-0736b675890a nodeName:}" failed. No retries permitted until 2026-07-28 16:44:52.553924029 +0000 UTC m=+1407.286553684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls") pod "sequence-graph-92dc3-8498bd67cd-n7qfz" (UID: "46788321-15b6-4aa8-ad9e-0736b675890a") : secret "sequence-graph-92dc3-serving-cert" not found
Jul 28 16:44:52.054342 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.054323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.556960 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.556930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.559337 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.559319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") pod \"sequence-graph-92dc3-8498bd67cd-n7qfz\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") " pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.744687 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.744660    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:52.865851 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:52.865828    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:44:52.868335 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:44:52.868309    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46788321_15b6_4aa8_ad9e_0736b675890a.slice/crio-63e6a2c46ced3e3edb673cc90deea02928b275b0e04dede840a17a2254dfb803 WatchSource:0}: Error finding container 63e6a2c46ced3e3edb673cc90deea02928b275b0e04dede840a17a2254dfb803: Status 404 returned error can't find the container with id 63e6a2c46ced3e3edb673cc90deea02928b275b0e04dede840a17a2254dfb803
Jul 28 16:44:53.807509 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:53.807477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" event={"ID":"46788321-15b6-4aa8-ad9e-0736b675890a","Type":"ContainerStarted","Data":"6cf8ee55bfd817b4935bd8428260ff843e78c6387455f578e56f208c9407003f"}
Jul 28 16:44:53.807678 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:53.807533    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" event={"ID":"46788321-15b6-4aa8-ad9e-0736b675890a","Type":"ContainerStarted","Data":"63e6a2c46ced3e3edb673cc90deea02928b275b0e04dede840a17a2254dfb803"}
Jul 28 16:44:53.807678 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:53.807575    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:44:53.822852 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:53.822808    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podStartSLOduration=2.822796781 podStartE2EDuration="2.822796781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:44:53.821533259 +0000 UTC m=+1408.554162936" watchObservedRunningTime="2026-07-28 16:44:53.822796781 +0000 UTC m=+1408.555426457"
Jul 28 16:44:59.817430 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:44:59.817396    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:45:01.930691 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:01.930610    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:45:01.931086 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:01.930817    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" containerID="cri-o://6cf8ee55bfd817b4935bd8428260ff843e78c6387455f578e56f208c9407003f" gracePeriod=30
Jul 28 16:45:02.150596 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.150560    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:45:02.150827 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.150795    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container" containerID="cri-o://fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5" gracePeriod=30
Jul 28 16:45:02.150977 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.150842    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kube-rbac-proxy" containerID="cri-o://81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3" gracePeriod=30
Jul 28 16:45:02.182185 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.182115    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:45:02.185553 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.185532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.188361 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.188343    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12f5e-predictor-serving-cert\""
Jul 28 16:45:02.188466 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.188346    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12f5e-kube-rbac-proxy-sar-config\""
Jul 28 16:45:02.196216 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.196198    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:45:02.199060 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.199039    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.199446 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.199421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:45:02.201252 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.201230    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c3541-kube-rbac-proxy-sar-config\""
Jul 28 16:45:02.201350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.201294    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c3541-serving-cert\""
Jul 28 16:45:02.207396 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.207378    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:45:02.329302 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.329270    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.329477 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.329316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.329477 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.329343    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjggw\" (UniqueName: \"kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.329477 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.329414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.329477 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.329450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.430327 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.430504 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.430504 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjggw\" (UniqueName: \"kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.430504 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430386    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.430504 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.430504 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:02.430476    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-serving-cert: secret "error-404-isvc-12f5e-predictor-serving-cert" not found
Jul 28 16:45:02.430728 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:02.430553    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls podName:8faf142d-7063-4055-880b-dea1c7a5851f nodeName:}" failed. No retries permitted until 2026-07-28 16:45:02.930532625 +0000 UTC m=+1417.663162301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls") pod "error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" (UID: "8faf142d-7063-4055-880b-dea1c7a5851f") : secret "error-404-isvc-12f5e-predictor-serving-cert" not found
Jul 28 16:45:02.430981 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.430962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.431095 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.431077    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.432888 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.432839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls\") pod \"ensemble-graph-c3541-95c57758-jf8c6\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") " pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.441900 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.441866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjggw\" (UniqueName: \"kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.508585 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.508558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.636830 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.636794    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:45:02.639670 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:45:02.639645    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38c11aa_4828_4aea_a1dc_390e58724ebe.slice/crio-758f035f771158f581cc2fa0cca02da297a8f63cc6ca3ab702e69ec24628c2ae WatchSource:0}: Error finding container 758f035f771158f581cc2fa0cca02da297a8f63cc6ca3ab702e69ec24628c2ae: Status 404 returned error can't find the container with id 758f035f771158f581cc2fa0cca02da297a8f63cc6ca3ab702e69ec24628c2ae
Jul 28 16:45:02.846321 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.846290    2555 generic.go:356] "Generic (PLEG): container finished" podID="f8c4522c-7798-426e-a390-5163faea5e7c" containerID="81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3" exitCode=2
Jul 28 16:45:02.846520 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.846361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerDied","Data":"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"}
Jul 28 16:45:02.847730 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.847701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" event={"ID":"a38c11aa-4828-4aea-a1dc-390e58724ebe","Type":"ContainerStarted","Data":"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"}
Jul 28 16:45:02.847872 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.847748    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" event={"ID":"a38c11aa-4828-4aea-a1dc-390e58724ebe","Type":"ContainerStarted","Data":"758f035f771158f581cc2fa0cca02da297a8f63cc6ca3ab702e69ec24628c2ae"}
Jul 28 16:45:02.847872 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.847808    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:02.864544 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.864507    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podStartSLOduration=0.864495858 podStartE2EDuration="864.495858ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:45:02.864234708 +0000 UTC m=+1417.596864385" watchObservedRunningTime="2026-07-28 16:45:02.864495858 +0000 UTC m=+1417.597125564"
Jul 28 16:45:02.933309 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:02.933275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:02.933679 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:02.933412    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-serving-cert: secret "error-404-isvc-12f5e-predictor-serving-cert" not found
Jul 28 16:45:02.933679 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:02.933464    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls podName:8faf142d-7063-4055-880b-dea1c7a5851f nodeName:}" failed. No retries permitted until 2026-07-28 16:45:03.933451177 +0000 UTC m=+1418.666080836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls") pod "error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" (UID: "8faf142d-7063-4055-880b-dea1c7a5851f") : secret "error-404-isvc-12f5e-predictor-serving-cert" not found
Jul 28 16:45:03.939805 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:03.939780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:03.942242 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:03.942222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") " pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:03.995303 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:03.995274    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:04.118336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.118302    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:45:04.121673 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:45:04.121650    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faf142d_7063_4055_880b_dea1c7a5851f.slice/crio-3512f5825d64345728201086cd670f85dc5ecb367c05f85e6cdcba21bd839ab0 WatchSource:0}: Error finding container 3512f5825d64345728201086cd670f85dc5ecb367c05f85e6cdcba21bd839ab0: Status 404 returned error can't find the container with id 3512f5825d64345728201086cd670f85dc5ecb367c05f85e6cdcba21bd839ab0
Jul 28 16:45:04.815711 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.815678    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:04.856576 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.856546    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerStarted","Data":"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"}
Jul 28 16:45:04.856762 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.856594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerStarted","Data":"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"}
Jul 28 16:45:04.856762 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.856614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerStarted","Data":"3512f5825d64345728201086cd670f85dc5ecb367c05f85e6cdcba21bd839ab0"}
Jul 28 16:45:04.856880 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.856834    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:04.856929 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.856886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:04.858043 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.858021    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:04.874646 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:04.874604    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podStartSLOduration=2.874590914 podStartE2EDuration="2.874590914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:45:04.873469772 +0000 UTC m=+1419.606099450" watchObservedRunningTime="2026-07-28 16:45:04.874590914 +0000 UTC m=+1419.607220629"
Jul 28 16:45:05.086617 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.086593    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:45:05.148985 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.148957    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74mnn\" (UniqueName: \"kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn\") pod \"f8c4522c-7798-426e-a390-5163faea5e7c\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") "
Jul 28 16:45:05.149145 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.149041    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls\") pod \"f8c4522c-7798-426e-a390-5163faea5e7c\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") "
Jul 28 16:45:05.149145 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.149070    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config\") pod \"f8c4522c-7798-426e-a390-5163faea5e7c\" (UID: \"f8c4522c-7798-426e-a390-5163faea5e7c\") "
Jul 28 16:45:05.149440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.149414    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8c4522c-7798-426e-a390-5163faea5e7c" (UID: "f8c4522c-7798-426e-a390-5163faea5e7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:45:05.151144 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.151119    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8c4522c-7798-426e-a390-5163faea5e7c" (UID: "f8c4522c-7798-426e-a390-5163faea5e7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:45:05.151228 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.151154    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn" (OuterVolumeSpecName: "kube-api-access-74mnn") pod "f8c4522c-7798-426e-a390-5163faea5e7c" (UID: "f8c4522c-7798-426e-a390-5163faea5e7c"). InnerVolumeSpecName "kube-api-access-74mnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:45:05.250231 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.250204    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c4522c-7798-426e-a390-5163faea5e7c-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:45:05.250231 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.250227    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c4522c-7798-426e-a390-5163faea5e7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:45:05.250354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.250239    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mnn\" (UniqueName: \"kubernetes.io/projected/f8c4522c-7798-426e-a390-5163faea5e7c-kube-api-access-74mnn\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:45:05.861227 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861195    2555 generic.go:356] "Generic (PLEG): container finished" podID="f8c4522c-7798-426e-a390-5163faea5e7c" containerID="fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5" exitCode=0
Jul 28 16:45:05.861336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861243    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"
Jul 28 16:45:05.861336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerDied","Data":"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"}
Jul 28 16:45:05.861336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9" event={"ID":"f8c4522c-7798-426e-a390-5163faea5e7c","Type":"ContainerDied","Data":"6926d7327f5fc048ba4a907453ca8922d11f2eb79a7608105f8b5b38392917a6"}
Jul 28 16:45:05.861448 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861358    2555 scope.go:117] "RemoveContainer" containerID="81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"
Jul 28 16:45:05.861857 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.861821    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:05.868949 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.868927    2555 scope.go:117] "RemoveContainer" containerID="fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"
Jul 28 16:45:05.875963 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.875950    2555 scope.go:117] "RemoveContainer" containerID="81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"
Jul 28 16:45:05.876237 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:05.876216    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3\": container with ID starting with 81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3 not found: ID does not exist" containerID="81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"
Jul 28 16:45:05.876312 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.876244    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3"} err="failed to get container status \"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3\": rpc error: code = NotFound desc = could not find container \"81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3\": container with ID starting with 81b8f2480a1ef600f88a55d4bc7acdf6560ec1445e7b62cebf21873e9cdeb6e3 not found: ID does not exist"
Jul 28 16:45:05.876312 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.876260    2555 scope.go:117] "RemoveContainer" containerID="fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"
Jul 28 16:45:05.876484 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:45:05.876466    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5\": container with ID starting with fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5 not found: ID does not exist" containerID="fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"
Jul 28 16:45:05.876530 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.876491    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5"} err="failed to get container status \"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5\": rpc error: code = NotFound desc = could not find container \"fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5\": container with ID starting with fb637c96e4cb63786818c417e2b458c863d02c5acaea3170bb7451e0154ec7f5 not found: ID does not exist"
Jul 28 16:45:05.887848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.887829    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:45:05.900246 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:05.900230    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92dc3-predictor-7f65f58dd-dz7g9"]
Jul 28 16:45:07.822883 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:07.822856    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" path="/var/lib/kubelet/pods/f8c4522c-7798-426e-a390-5163faea5e7c/volumes"
Jul 28 16:45:08.856713 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:08.856685    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:45:09.815016 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:09.814976    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:10.865939 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:10.865908    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:45:10.866375 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:10.866300    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:14.815467 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:14.815431    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:14.815816 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:14.815532    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:45:19.815254 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:19.815178    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:20.866840 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:20.866799    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:24.815431 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:24.815395    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:29.815193 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:29.815157    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:45:30.866456 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:30.866415    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:31.959026 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:31.958972    2555 generic.go:356] "Generic (PLEG): container finished" podID="46788321-15b6-4aa8-ad9e-0736b675890a" containerID="6cf8ee55bfd817b4935bd8428260ff843e78c6387455f578e56f208c9407003f" exitCode=0
Jul 28 16:45:31.959297 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:31.959036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" event={"ID":"46788321-15b6-4aa8-ad9e-0736b675890a","Type":"ContainerDied","Data":"6cf8ee55bfd817b4935bd8428260ff843e78c6387455f578e56f208c9407003f"}
Jul 28 16:45:32.068789 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.068768    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:45:32.237367 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.237294    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle\") pod \"46788321-15b6-4aa8-ad9e-0736b675890a\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") "
Jul 28 16:45:32.237367 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.237334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") pod \"46788321-15b6-4aa8-ad9e-0736b675890a\" (UID: \"46788321-15b6-4aa8-ad9e-0736b675890a\") "
Jul 28 16:45:32.237661 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.237636    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "46788321-15b6-4aa8-ad9e-0736b675890a" (UID: "46788321-15b6-4aa8-ad9e-0736b675890a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:45:32.239473 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.239448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46788321-15b6-4aa8-ad9e-0736b675890a" (UID: "46788321-15b6-4aa8-ad9e-0736b675890a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:45:32.338432 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.338407    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46788321-15b6-4aa8-ad9e-0736b675890a-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.338432 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.338430    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46788321-15b6-4aa8-ad9e-0736b675890a-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.963149 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.963125    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"
Jul 28 16:45:32.963541 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.963155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz" event={"ID":"46788321-15b6-4aa8-ad9e-0736b675890a","Type":"ContainerDied","Data":"63e6a2c46ced3e3edb673cc90deea02928b275b0e04dede840a17a2254dfb803"}
Jul 28 16:45:32.963541 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.963204    2555 scope.go:117] "RemoveContainer" containerID="6cf8ee55bfd817b4935bd8428260ff843e78c6387455f578e56f208c9407003f"
Jul 28 16:45:32.988073 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.988047    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:45:32.991629 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:32.991610    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92dc3-8498bd67cd-n7qfz"]
Jul 28 16:45:33.822975 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:33.822943    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" path="/var/lib/kubelet/pods/46788321-15b6-4aa8-ad9e-0736b675890a/volumes"
Jul 28 16:45:40.866968 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:40.866925    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:50.867596 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:45:50.867568    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:46:02.173398 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173365    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173636    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173645    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173653    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173658    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173676    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kube-rbac-proxy"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173682    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kube-rbac-proxy"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173721    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kube-rbac-proxy"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173728    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="46788321-15b6-4aa8-ad9e-0736b675890a" containerName="sequence-graph-92dc3"
Jul 28 16:46:02.173809 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.173738    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c4522c-7798-426e-a390-5163faea5e7c" containerName="kserve-container"
Jul 28 16:46:02.176446 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.176427    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.178845 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.178821    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-12f5e-kube-rbac-proxy-sar-config\""
Jul 28 16:46:02.179038 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.179023    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-12f5e-serving-cert\""
Jul 28 16:46:02.186788 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.186767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:46:02.341471 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.341440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.341597 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.341486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.442614 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.442555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.442614 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.442599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.442758 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:46:02.442692    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-12f5e-serving-cert: secret "sequence-graph-12f5e-serving-cert" not found
Jul 28 16:46:02.442758 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:46:02.442752    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls podName:022135ec-1bbe-4c18-84a9-3643ca0edf64 nodeName:}" failed. No retries permitted until 2026-07-28 16:46:02.94273733 +0000 UTC m=+1477.675366989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls") pod "sequence-graph-12f5e-77974c8875-46cwl" (UID: "022135ec-1bbe-4c18-84a9-3643ca0edf64") : secret "sequence-graph-12f5e-serving-cert" not found
Jul 28 16:46:02.443171 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.443157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.947285 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.947249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:02.949611 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:02.949594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") pod \"sequence-graph-12f5e-77974c8875-46cwl\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") " pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:03.086216 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:03.086194    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:03.208624 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:03.208598    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:46:03.211190 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:46:03.211164    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022135ec_1bbe_4c18_84a9_3643ca0edf64.slice/crio-a602f7558e1daae041a5256b1866e34459e366973104b97caec045f519c872f4 WatchSource:0}: Error finding container a602f7558e1daae041a5256b1866e34459e366973104b97caec045f519c872f4: Status 404 returned error can't find the container with id a602f7558e1daae041a5256b1866e34459e366973104b97caec045f519c872f4
Jul 28 16:46:04.072749 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:04.072716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" event={"ID":"022135ec-1bbe-4c18-84a9-3643ca0edf64","Type":"ContainerStarted","Data":"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"}
Jul 28 16:46:04.072900 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:04.072770    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" event={"ID":"022135ec-1bbe-4c18-84a9-3643ca0edf64","Type":"ContainerStarted","Data":"a602f7558e1daae041a5256b1866e34459e366973104b97caec045f519c872f4"}
Jul 28 16:46:04.072900 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:04.072803    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:04.091947 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:04.091906    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podStartSLOduration=2.091892515 podStartE2EDuration="2.091892515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:46:04.090255143 +0000 UTC m=+1478.822884819" watchObservedRunningTime="2026-07-28 16:46:04.091892515 +0000 UTC m=+1478.824522191"
Jul 28 16:46:10.081119 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:10.081088    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:46:25.700307 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:25.700272    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:46:25.705557 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:46:25.705536    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:51:25.720713 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:51:25.720683    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:51:25.726486 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:51:25.726467    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:53:16.834702 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:16.834669    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:53:16.837574 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:16.834852    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" containerID="cri-o://2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9" gracePeriod=30
Jul 28 16:53:17.058585 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.058552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:53:17.058832 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.058809    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" containerID="cri-o://b8aaf862aad6446fbcba17b549e0a7f176db782e6eaa039e155fbfae80935258" gracePeriod=30
Jul 28 16:53:17.058965 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.058901    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kube-rbac-proxy" containerID="cri-o://8982ff89ee0a1e050645dda3ef88189679452b81a1e55e8a719c6897a5876833" gracePeriod=30
Jul 28 16:53:17.090556 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.090500    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:53:17.093730 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.093705    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.095847 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.095830    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2717b-kube-rbac-proxy-sar-config\""
Jul 28 16:53:17.095920 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.095857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2717b-predictor-serving-cert\""
Jul 28 16:53:17.102284 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.102264    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:53:17.159339 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.159316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.159450 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.159354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.159450 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.159370    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2fh\" (UniqueName: \"kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.260284 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.260257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.260410 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.260289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2fh\" (UniqueName: \"kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.260410 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.260348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.260839 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.260820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.262659 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.262640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.270986 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.268520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2fh\" (UniqueName: \"kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh\") pod \"error-404-isvc-2717b-predictor-5c5b975b48-gtnpv\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") " pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.404181 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.404123    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:17.528068 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.527943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:53:17.530652 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:53:17.530630    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c5545c_58dd_4a4a_b965_cfbb88ac7ba0.slice/crio-f722472fd245913d82a4a4c7cec510fdd9d67902dec2b93387988710d7e1147b WatchSource:0}: Error finding container f722472fd245913d82a4a4c7cec510fdd9d67902dec2b93387988710d7e1147b: Status 404 returned error can't find the container with id f722472fd245913d82a4a4c7cec510fdd9d67902dec2b93387988710d7e1147b
Jul 28 16:53:17.581324 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.581292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerStarted","Data":"f722472fd245913d82a4a4c7cec510fdd9d67902dec2b93387988710d7e1147b"}
Jul 28 16:53:17.583182 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.583160    2555 generic.go:356] "Generic (PLEG): container finished" podID="b2edb5f1-1799-4f12-af81-4659496e8837" containerID="8982ff89ee0a1e050645dda3ef88189679452b81a1e55e8a719c6897a5876833" exitCode=2
Jul 28 16:53:17.583283 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:17.583188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerDied","Data":"8982ff89ee0a1e050645dda3ef88189679452b81a1e55e8a719c6897a5876833"}
Jul 28 16:53:18.588336 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.588301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerStarted","Data":"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"}
Jul 28 16:53:18.588696 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.588361    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:18.588696 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.588382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerStarted","Data":"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"}
Jul 28 16:53:18.588696 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.588500    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:18.589714 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.589687    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:53:18.605907 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.605867    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podStartSLOduration=1.605855076 podStartE2EDuration="1.605855076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:53:18.604549911 +0000 UTC m=+1913.337179588" watchObservedRunningTime="2026-07-28 16:53:18.605855076 +0000 UTC m=+1913.338484752"
Jul 28 16:53:18.855934 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:18.855852    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:19.594138 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.594114    2555 generic.go:356] "Generic (PLEG): container finished" podID="b2edb5f1-1799-4f12-af81-4659496e8837" containerID="b8aaf862aad6446fbcba17b549e0a7f176db782e6eaa039e155fbfae80935258" exitCode=0
Jul 28 16:53:19.594439 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.594182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerDied","Data":"b8aaf862aad6446fbcba17b549e0a7f176db782e6eaa039e155fbfae80935258"}
Jul 28 16:53:19.594524 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.594499    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:53:19.618565 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.618531    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:53:19.623472 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.623446    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:53:19.694652 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.694627    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:53:19.779270 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.779239    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls\") pod \"b2edb5f1-1799-4f12-af81-4659496e8837\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") "
Jul 28 16:53:19.779414 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.779282    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk8hd\" (UniqueName: \"kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd\") pod \"b2edb5f1-1799-4f12-af81-4659496e8837\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") "
Jul 28 16:53:19.779414 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.779332    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config\") pod \"b2edb5f1-1799-4f12-af81-4659496e8837\" (UID: \"b2edb5f1-1799-4f12-af81-4659496e8837\") "
Jul 28 16:53:19.779665 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.779643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2edb5f1-1799-4f12-af81-4659496e8837" (UID: "b2edb5f1-1799-4f12-af81-4659496e8837"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:53:19.781388 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.781359    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2edb5f1-1799-4f12-af81-4659496e8837" (UID: "b2edb5f1-1799-4f12-af81-4659496e8837"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:53:19.781485 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.781388    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd" (OuterVolumeSpecName: "kube-api-access-zk8hd") pod "b2edb5f1-1799-4f12-af81-4659496e8837" (UID: "b2edb5f1-1799-4f12-af81-4659496e8837"). InnerVolumeSpecName "kube-api-access-zk8hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:53:19.880076 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.880052    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2edb5f1-1799-4f12-af81-4659496e8837-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:53:19.880076 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.880074    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2edb5f1-1799-4f12-af81-4659496e8837-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:53:19.880216 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:19.880085    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8hd\" (UniqueName: \"kubernetes.io/projected/b2edb5f1-1799-4f12-af81-4659496e8837-kube-api-access-zk8hd\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:53:20.599288 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.599257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq" event={"ID":"b2edb5f1-1799-4f12-af81-4659496e8837","Type":"ContainerDied","Data":"3b0343492c72debb776352e7e4efa8a557eedb23a6f6ab0e977afe337c655207"}
Jul 28 16:53:20.599725 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.599321    2555 scope.go:117] "RemoveContainer" containerID="8982ff89ee0a1e050645dda3ef88189679452b81a1e55e8a719c6897a5876833"
Jul 28 16:53:20.599725 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.599267    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"
Jul 28 16:53:20.606992 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.606971    2555 scope.go:117] "RemoveContainer" containerID="b8aaf862aad6446fbcba17b549e0a7f176db782e6eaa039e155fbfae80935258"
Jul 28 16:53:20.615890 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.615870    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:53:20.619513 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:20.619494    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3541-predictor-67c685758b-jbrfq"]
Jul 28 16:53:21.822988 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:21.822960    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" path="/var/lib/kubelet/pods/b2edb5f1-1799-4f12-af81-4659496e8837/volumes"
Jul 28 16:53:23.856125 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:23.856088    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:24.600069 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:24.600036    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:53:24.600611 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:24.600582    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:53:28.855878 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:28.855825    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:28.856329 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:28.855961    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:53:33.855212 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:33.855165    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:34.601232 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:34.601193    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:53:38.856204 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:38.856168    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:43.854893 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:43.854858    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:53:44.600706 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:44.600673    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:53:47.480077 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.480054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:53:47.572269 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.572234    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls\") pod \"a38c11aa-4828-4aea-a1dc-390e58724ebe\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") "
Jul 28 16:53:47.572429 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.572283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle\") pod \"a38c11aa-4828-4aea-a1dc-390e58724ebe\" (UID: \"a38c11aa-4828-4aea-a1dc-390e58724ebe\") "
Jul 28 16:53:47.572618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.572595    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a38c11aa-4828-4aea-a1dc-390e58724ebe" (UID: "a38c11aa-4828-4aea-a1dc-390e58724ebe"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:53:47.574421 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.574393    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a38c11aa-4828-4aea-a1dc-390e58724ebe" (UID: "a38c11aa-4828-4aea-a1dc-390e58724ebe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:53:47.673058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.672985    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38c11aa-4828-4aea-a1dc-390e58724ebe-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:53:47.673058 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.673025    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38c11aa-4828-4aea-a1dc-390e58724ebe-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:53:47.697935 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.697906    2555 generic.go:356] "Generic (PLEG): container finished" podID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerID="2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9" exitCode=0
Jul 28 16:53:47.698101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.697943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" event={"ID":"a38c11aa-4828-4aea-a1dc-390e58724ebe","Type":"ContainerDied","Data":"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"}
Jul 28 16:53:47.698101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.697958    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"
Jul 28 16:53:47.698101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.697984    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6" event={"ID":"a38c11aa-4828-4aea-a1dc-390e58724ebe","Type":"ContainerDied","Data":"758f035f771158f581cc2fa0cca02da297a8f63cc6ca3ab702e69ec24628c2ae"}
Jul 28 16:53:47.698101 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.698029    2555 scope.go:117] "RemoveContainer" containerID="2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"
Jul 28 16:53:47.705929 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.705914    2555 scope.go:117] "RemoveContainer" containerID="2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"
Jul 28 16:53:47.706173 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:53:47.706154    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9\": container with ID starting with 2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9 not found: ID does not exist" containerID="2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"
Jul 28 16:53:47.706230 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.706180    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9"} err="failed to get container status \"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9\": rpc error: code = NotFound desc = could not find container \"2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9\": container with ID starting with 2ab6497f1600debaa2673823ab7e493ef84aaede8af2d205bcd6d78375b127a9 not found: ID does not exist"
Jul 28 16:53:47.716409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.716387    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:53:47.722748 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.722729    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c3541-95c57758-jf8c6"]
Jul 28 16:53:47.824327 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:47.824294    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" path="/var/lib/kubelet/pods/a38c11aa-4828-4aea-a1dc-390e58724ebe/volumes"
Jul 28 16:53:54.601484 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:53:54.601448    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:54:04.601158 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:04.601130    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:54:16.698584 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.698487    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:54:16.699133 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.698701    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" containerID="cri-o://518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709" gracePeriod=30
Jul 28 16:54:16.856904 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.856870    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:54:16.857135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.857110    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container" containerID="cri-o://fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18" gracePeriod=30
Jul 28 16:54:16.857211 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.857159    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kube-rbac-proxy" containerID="cri-o://0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885" gracePeriod=30
Jul 28 16:54:16.972119 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972052    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 16:54:16.972382 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972361    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541"
Jul 28 16:54:16.972382 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972382    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972397    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972403    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972411    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kube-rbac-proxy"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972417    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kube-rbac-proxy"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972468    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38c11aa-4828-4aea-a1dc-390e58724ebe" containerName="ensemble-graph-c3541"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972475    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kserve-container"
Jul 28 16:54:16.972558 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.972483    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2edb5f1-1799-4f12-af81-4659496e8837" containerName="kube-rbac-proxy"
Jul 28 16:54:16.975573 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.975556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:16.977914 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.977889    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b9a8c-kube-rbac-proxy-sar-config\""
Jul 28 16:54:16.978027 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.977890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b9a8c-predictor-serving-cert\""
Jul 28 16:54:16.983969 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:16.983950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 16:54:17.070531 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.070499    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:17.073767 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.073747    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.076126 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.076101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2717b-kube-rbac-proxy-sar-config\""
Jul 28 16:54:17.076224 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.076106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2717b-serving-cert\""
Jul 28 16:54:17.082500 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.082481    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:17.090055 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.090036    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.090171 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.090066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zjd\" (UniqueName: \"kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.090171 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.090090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.191138 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.191292 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.191292 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191188    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zjd\" (UniqueName: \"kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.191292 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.191292 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191235    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.191292 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:17.191271    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-serving-cert: secret "error-404-isvc-b9a8c-predictor-serving-cert" not found
Jul 28 16:54:17.191624 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:17.191348    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls podName:e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2 nodeName:}" failed. No retries permitted until 2026-07-28 16:54:17.69132972 +0000 UTC m=+1972.423959375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls") pod "error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" (UID: "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2") : secret "error-404-isvc-b9a8c-predictor-serving-cert" not found
Jul 28 16:54:17.191907 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.191886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.199126 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.199106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zjd\" (UniqueName: \"kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.292176 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.292150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.292286 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.292203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.292752 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.292735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.294469 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.294451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls\") pod \"splitter-graph-2717b-5fd8b855cc-sldff\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") " pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.384470 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.384445    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.506428 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.506348    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:17.508882 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:54:17.508858    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4cf707_e78a_4ca5_ab2c_09f0388d2f22.slice/crio-c75bc2071f41405a25414101fd60bfc090428f79979fd4bf99f14b2313a85821 WatchSource:0}: Error finding container c75bc2071f41405a25414101fd60bfc090428f79979fd4bf99f14b2313a85821: Status 404 returned error can't find the container with id c75bc2071f41405a25414101fd60bfc090428f79979fd4bf99f14b2313a85821
Jul 28 16:54:17.695893 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.695858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.698292 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.698274    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") pod \"error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") " pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:17.801130 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.801047    2555 generic.go:356] "Generic (PLEG): container finished" podID="8faf142d-7063-4055-880b-dea1c7a5851f" containerID="0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885" exitCode=2
Jul 28 16:54:17.801130 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.801112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerDied","Data":"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"}
Jul 28 16:54:17.802516 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.802495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" event={"ID":"be4cf707-e78a-4ca5-ab2c-09f0388d2f22","Type":"ContainerStarted","Data":"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"}
Jul 28 16:54:17.802632 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.802534    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" event={"ID":"be4cf707-e78a-4ca5-ab2c-09f0388d2f22","Type":"ContainerStarted","Data":"c75bc2071f41405a25414101fd60bfc090428f79979fd4bf99f14b2313a85821"}
Jul 28 16:54:17.802687 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.802633    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:17.816451 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.816410    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podStartSLOduration=0.816396606 podStartE2EDuration="816.396606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:54:17.816136465 +0000 UTC m=+1972.548766143" watchObservedRunningTime="2026-07-28 16:54:17.816396606 +0000 UTC m=+1972.549026261"
Jul 28 16:54:17.885908 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:17.885876    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:18.012521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:18.012488    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 16:54:18.015364 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:54:18.015335    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85b47f1_f9d8_46f3_8d95_ae9e1c2997b2.slice/crio-e5a393d974e9d056654dcb444be057427e8edc9ba62278eb74fc2cd46891811f WatchSource:0}: Error finding container e5a393d974e9d056654dcb444be057427e8edc9ba62278eb74fc2cd46891811f: Status 404 returned error can't find the container with id e5a393d974e9d056654dcb444be057427e8edc9ba62278eb74fc2cd46891811f
Jul 28 16:54:18.807963 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:18.807930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerStarted","Data":"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"}
Jul 28 16:54:18.808354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:18.807985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerStarted","Data":"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"}
Jul 28 16:54:18.808354 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:18.808029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerStarted","Data":"e5a393d974e9d056654dcb444be057427e8edc9ba62278eb74fc2cd46891811f"}
Jul 28 16:54:18.824828 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:18.824782    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podStartSLOduration=2.824766187 podStartE2EDuration="2.824766187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:54:18.822886041 +0000 UTC m=+1973.555515717" watchObservedRunningTime="2026-07-28 16:54:18.824766187 +0000 UTC m=+1973.557395862"
Jul 28 16:54:19.689561 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.689539    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:54:19.812347 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.812272    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") pod \"8faf142d-7063-4055-880b-dea1c7a5851f\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") "
Jul 28 16:54:19.812347 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.812310    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjggw\" (UniqueName: \"kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw\") pod \"8faf142d-7063-4055-880b-dea1c7a5851f\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") "
Jul 28 16:54:19.812347 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.812347    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config\") pod \"8faf142d-7063-4055-880b-dea1c7a5851f\" (UID: \"8faf142d-7063-4055-880b-dea1c7a5851f\") "
Jul 28 16:54:19.812885 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.812780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8faf142d-7063-4055-880b-dea1c7a5851f" (UID: "8faf142d-7063-4055-880b-dea1c7a5851f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:19.814203 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814175    2555 generic.go:356] "Generic (PLEG): container finished" podID="8faf142d-7063-4055-880b-dea1c7a5851f" containerID="fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18" exitCode=0
Jul 28 16:54:19.814333 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814251    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"
Jul 28 16:54:19.814333 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerDied","Data":"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"}
Jul 28 16:54:19.814333 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2" event={"ID":"8faf142d-7063-4055-880b-dea1c7a5851f","Type":"ContainerDied","Data":"3512f5825d64345728201086cd670f85dc5ecb367c05f85e6cdcba21bd839ab0"}
Jul 28 16:54:19.814475 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814370    2555 scope.go:117] "RemoveContainer" containerID="0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"
Jul 28 16:54:19.814818 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814728    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8faf142d-7063-4055-880b-dea1c7a5851f" (UID: "8faf142d-7063-4055-880b-dea1c7a5851f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:54:19.814818 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814763    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw" (OuterVolumeSpecName: "kube-api-access-jjggw") pod "8faf142d-7063-4055-880b-dea1c7a5851f" (UID: "8faf142d-7063-4055-880b-dea1c7a5851f"). InnerVolumeSpecName "kube-api-access-jjggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:54:19.814818 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:19.814987 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.814823    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:19.815836 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.815815    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:19.825410 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.825398    2555 scope.go:117] "RemoveContainer" containerID="fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"
Jul 28 16:54:19.832600 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.832580    2555 scope.go:117] "RemoveContainer" containerID="0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"
Jul 28 16:54:19.832842 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:19.832825    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885\": container with ID starting with 0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885 not found: ID does not exist" containerID="0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"
Jul 28 16:54:19.832915 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.832854    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885"} err="failed to get container status \"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885\": rpc error: code = NotFound desc = could not find container \"0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885\": container with ID starting with 0c77be96ac87aebe14a4c879261c8d08034b5c33782a5773cae47ea740310885 not found: ID does not exist"
Jul 28 16:54:19.832915 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.832875    2555 scope.go:117] "RemoveContainer" containerID="fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"
Jul 28 16:54:19.833127 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:19.833110    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18\": container with ID starting with fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18 not found: ID does not exist" containerID="fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"
Jul 28 16:54:19.833169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.833132    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18"} err="failed to get container status \"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18\": rpc error: code = NotFound desc = could not find container \"fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18\": container with ID starting with fc1ed0d9dda550315a88b9e057eea32ea2dc1b02e7663852bb6d8b01a1f69f18 not found: ID does not exist"
Jul 28 16:54:19.913649 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.913626    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8faf142d-7063-4055-880b-dea1c7a5851f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:19.913649 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.913646    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8faf142d-7063-4055-880b-dea1c7a5851f-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:19.913771 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:19.913657    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjggw\" (UniqueName: \"kubernetes.io/projected/8faf142d-7063-4055-880b-dea1c7a5851f-kube-api-access-jjggw\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:20.080105 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:20.080027    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:20.128791 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:20.128765    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:54:20.132833 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:20.132805    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12f5e-predictor-79d5b64b64-5qnk2"]
Jul 28 16:54:20.818937 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:20.818894    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:21.823676 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:21.823642    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" path="/var/lib/kubelet/pods/8faf142d-7063-4055-880b-dea1c7a5851f/volumes"
Jul 28 16:54:23.813560 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:23.813531    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:25.079377 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:25.079329    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:25.824176 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:25.824145    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:54:25.824724 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:25.824699    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:27.128475 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.128391    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:27.128903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.128617    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" containerID="cri-o://83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc" gracePeriod=30
Jul 28 16:54:27.289443 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.289407    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:54:27.289637 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.289616    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container" containerID="cri-o://c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65" gracePeriod=30
Jul 28 16:54:27.289811 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.289760    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kube-rbac-proxy" containerID="cri-o://8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b" gracePeriod=30
Jul 28 16:54:27.395717 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.395633    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 16:54:27.396035 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396019    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kube-rbac-proxy"
Jul 28 16:54:27.396123 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396038    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kube-rbac-proxy"
Jul 28 16:54:27.396123 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396052    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container"
Jul 28 16:54:27.396123 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396059    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container"
Jul 28 16:54:27.396269 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396134    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kserve-container"
Jul 28 16:54:27.396269 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.396146    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8faf142d-7063-4055-880b-dea1c7a5851f" containerName="kube-rbac-proxy"
Jul 28 16:54:27.399024 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.398979    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.401943 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.401921    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6e34-predictor-serving-cert\""
Jul 28 16:54:27.402255 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.402238    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6e34-kube-rbac-proxy-sar-config\""
Jul 28 16:54:27.419901 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.419877    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 16:54:27.565387 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.565354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.565387 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.565392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.565590 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.565417    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj56j\" (UniqueName: \"kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.666756 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.666667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.666756 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.666714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.666756 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.666751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj56j\" (UniqueName: \"kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.667435 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.667413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.669414 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.669393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.674889 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.674859    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj56j\" (UniqueName: \"kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j\") pod \"error-404-isvc-b6e34-predictor-d4bb678f4-x429j\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") " pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.709685 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.709649    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:27.842012 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.841972    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 16:54:27.844653 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:54:27.844625    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689c2464_57c6_4a45_8ab0_4de108999c77.slice/crio-5c14377325fca4a1611691b442db71db8abe2eab8f600fedcfd8c4a3a43568e6 WatchSource:0}: Error finding container 5c14377325fca4a1611691b442db71db8abe2eab8f600fedcfd8c4a3a43568e6: Status 404 returned error can't find the container with id 5c14377325fca4a1611691b442db71db8abe2eab8f600fedcfd8c4a3a43568e6
Jul 28 16:54:27.847309 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.847279    2555 generic.go:356] "Generic (PLEG): container finished" podID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerID="8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b" exitCode=2
Jul 28 16:54:27.847415 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:27.847323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerDied","Data":"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"}
Jul 28 16:54:28.812777 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.812731    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:28.856970 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.856932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerStarted","Data":"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"}
Jul 28 16:54:28.857169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.857042    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:28.857169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.857068    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:28.857169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.857085    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerStarted","Data":"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"}
Jul 28 16:54:28.857169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.857111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerStarted","Data":"5c14377325fca4a1611691b442db71db8abe2eab8f600fedcfd8c4a3a43568e6"}
Jul 28 16:54:28.858246 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.858218    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:54:28.875557 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:28.875510    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podStartSLOduration=1.8754967329999999 podStartE2EDuration="1.875496733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:54:28.87386472 +0000 UTC m=+1983.606494398" watchObservedRunningTime="2026-07-28 16:54:28.875496733 +0000 UTC m=+1983.608126408"
Jul 28 16:54:29.595515 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:29.595479    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:54:29.860996 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:29.860906    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:54:30.079541 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.079507    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:30.079684 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.079596    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:54:30.335258 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.335237    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:54:30.489676 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.489604    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config\") pod \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") "
Jul 28 16:54:30.489676 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.489653    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm2fh\" (UniqueName: \"kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh\") pod \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") "
Jul 28 16:54:30.489846 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.489703    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls\") pod \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\" (UID: \"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0\") "
Jul 28 16:54:30.490064 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.490026    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" (UID: "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:30.491802 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.491782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" (UID: "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:54:30.491895 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.491841    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh" (OuterVolumeSpecName: "kube-api-access-jm2fh") pod "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" (UID: "a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0"). InnerVolumeSpecName "kube-api-access-jm2fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:54:30.591154 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.591131    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2fh\" (UniqueName: \"kubernetes.io/projected/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-api-access-jm2fh\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:30.591154 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.591152    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:30.591300 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.591162    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:30.865681 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.865640    2555 generic.go:356] "Generic (PLEG): container finished" podID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerID="c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65" exitCode=0
Jul 28 16:54:30.866169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.865696    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"
Jul 28 16:54:30.866169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.865704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerDied","Data":"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"}
Jul 28 16:54:30.866169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.865778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv" event={"ID":"a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0","Type":"ContainerDied","Data":"f722472fd245913d82a4a4c7cec510fdd9d67902dec2b93387988710d7e1147b"}
Jul 28 16:54:30.866169 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.865820    2555 scope.go:117] "RemoveContainer" containerID="8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"
Jul 28 16:54:30.874474 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.874459    2555 scope.go:117] "RemoveContainer" containerID="c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"
Jul 28 16:54:30.881266 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.881251    2555 scope.go:117] "RemoveContainer" containerID="8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"
Jul 28 16:54:30.881516 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:30.881498    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b\": container with ID starting with 8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b not found: ID does not exist" containerID="8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"
Jul 28 16:54:30.881578 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.881525    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b"} err="failed to get container status \"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b\": rpc error: code = NotFound desc = could not find container \"8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b\": container with ID starting with 8304639ac082483dfdbda4ab9e33fbcb965e4395b4bf47a7a039b775136b8e8b not found: ID does not exist"
Jul 28 16:54:30.881578 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.881542    2555 scope.go:117] "RemoveContainer" containerID="c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"
Jul 28 16:54:30.881772 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:30.881753    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65\": container with ID starting with c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65 not found: ID does not exist" containerID="c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"
Jul 28 16:54:30.881812 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.881775    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65"} err="failed to get container status \"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65\": rpc error: code = NotFound desc = could not find container \"c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65\": container with ID starting with c12276316e46ea8167c81c615aa8406ab362c3c301c35ba20dc5e9b90f2aba65 not found: ID does not exist"
Jul 28 16:54:30.886626 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.886606    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:54:30.890256 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:30.890239    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2717b-predictor-5c5b975b48-gtnpv"]
Jul 28 16:54:31.823204 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:31.823173    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" path="/var/lib/kubelet/pods/a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0/volumes"
Jul 28 16:54:33.812448 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:33.812406    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:34.866667 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:34.866638    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:54:34.867133 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:34.867107    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:54:35.080568 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:35.080531    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:35.825174 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:35.825136    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:38.812506 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:38.812464    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:38.812928 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:38.812574    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:40.079865 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:40.079831    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:43.812713 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:43.812666    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:44.867826 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:44.867793    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:54:45.079553 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:45.079517    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:45.824768 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:45.824734    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:46.841593 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.841570    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:54:46.909746 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.909713    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") pod \"022135ec-1bbe-4c18-84a9-3643ca0edf64\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") "
Jul 28 16:54:46.909915 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.909760    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle\") pod \"022135ec-1bbe-4c18-84a9-3643ca0edf64\" (UID: \"022135ec-1bbe-4c18-84a9-3643ca0edf64\") "
Jul 28 16:54:46.910131 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.910108    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "022135ec-1bbe-4c18-84a9-3643ca0edf64" (UID: "022135ec-1bbe-4c18-84a9-3643ca0edf64"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:46.911860 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.911831    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "022135ec-1bbe-4c18-84a9-3643ca0edf64" (UID: "022135ec-1bbe-4c18-84a9-3643ca0edf64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:54:46.921737 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.921713    2555 generic.go:356] "Generic (PLEG): container finished" podID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerID="518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709" exitCode=0
Jul 28 16:54:46.921834 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.921749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" event={"ID":"022135ec-1bbe-4c18-84a9-3643ca0edf64","Type":"ContainerDied","Data":"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"}
Jul 28 16:54:46.921834 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.921766    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"
Jul 28 16:54:46.921946 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.921849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl" event={"ID":"022135ec-1bbe-4c18-84a9-3643ca0edf64","Type":"ContainerDied","Data":"a602f7558e1daae041a5256b1866e34459e366973104b97caec045f519c872f4"}
Jul 28 16:54:46.921946 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.921885    2555 scope.go:117] "RemoveContainer" containerID="518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"
Jul 28 16:54:46.929729 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.929715    2555 scope.go:117] "RemoveContainer" containerID="518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"
Jul 28 16:54:46.929985 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:46.929968    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709\": container with ID starting with 518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709 not found: ID does not exist" containerID="518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"
Jul 28 16:54:46.930051 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.929991    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709"} err="failed to get container status \"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709\": rpc error: code = NotFound desc = could not find container \"518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709\": container with ID starting with 518d4032316121c1af0eecdc6cde63b1a0b4c12d82122b2c2c10860cdf488709 not found: ID does not exist"
Jul 28 16:54:46.940899 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.940881    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:54:46.944694 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:46.944675    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-12f5e-77974c8875-46cwl"]
Jul 28 16:54:47.010102 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:47.010077    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022135ec-1bbe-4c18-84a9-3643ca0edf64-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:47.010102 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:47.010097    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/022135ec-1bbe-4c18-84a9-3643ca0edf64-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:47.823350 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:47.823318    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" path="/var/lib/kubelet/pods/022135ec-1bbe-4c18-84a9-3643ca0edf64/volumes"
Jul 28 16:54:48.812150 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:48.812118    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:53.812440 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:53.812402    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:54:54.867455 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:54.867421    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:54:55.825519 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:55.825481    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:54:57.270135 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.270112    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:57.278749 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.278730    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle\") pod \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") "
Jul 28 16:54:57.278835 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.278763    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls\") pod \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\" (UID: \"be4cf707-e78a-4ca5-ab2c-09f0388d2f22\") "
Jul 28 16:54:57.279051 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.279030    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "be4cf707-e78a-4ca5-ab2c-09f0388d2f22" (UID: "be4cf707-e78a-4ca5-ab2c-09f0388d2f22"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:57.280782 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.280754    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be4cf707-e78a-4ca5-ab2c-09f0388d2f22" (UID: "be4cf707-e78a-4ca5-ab2c-09f0388d2f22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:54:57.379284 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.379253    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:57.379284 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.379282    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4cf707-e78a-4ca5-ab2c-09f0388d2f22-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 16:54:57.961967 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.961926    2555 generic.go:356] "Generic (PLEG): container finished" podID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerID="83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc" exitCode=0
Jul 28 16:54:57.962155 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.961976    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"
Jul 28 16:54:57.962155 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.962023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" event={"ID":"be4cf707-e78a-4ca5-ab2c-09f0388d2f22","Type":"ContainerDied","Data":"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"}
Jul 28 16:54:57.962155 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.962076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff" event={"ID":"be4cf707-e78a-4ca5-ab2c-09f0388d2f22","Type":"ContainerDied","Data":"c75bc2071f41405a25414101fd60bfc090428f79979fd4bf99f14b2313a85821"}
Jul 28 16:54:57.962155 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.962099    2555 scope.go:117] "RemoveContainer" containerID="83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"
Jul 28 16:54:57.970201 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.970182    2555 scope.go:117] "RemoveContainer" containerID="83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"
Jul 28 16:54:57.970477 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:54:57.970458    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc\": container with ID starting with 83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc not found: ID does not exist" containerID="83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"
Jul 28 16:54:57.970527 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.970488    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc"} err="failed to get container status \"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc\": rpc error: code = NotFound desc = could not find container \"83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc\": container with ID starting with 83a8886717f442cfad3b31857bfff62e067fb46bafc1e0dcb34de24c8216a0cc not found: ID does not exist"
Jul 28 16:54:57.979611 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.979582    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:57.983190 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:57.983170    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2717b-5fd8b855cc-sldff"]
Jul 28 16:54:59.827618 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:54:59.827586    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" path="/var/lib/kubelet/pods/be4cf707-e78a-4ca5-ab2c-09f0388d2f22/volumes"
Jul 28 16:55:04.867170 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:04.867137    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:55:05.825419 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:05.825393    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 16:55:14.868096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:14.868062    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 16:55:16.921224 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921194    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921468    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921478    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921485    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kube-rbac-proxy"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921491    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kube-rbac-proxy"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921502    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921507    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921515    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921520    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921565    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="022135ec-1bbe-4c18-84a9-3643ca0edf64" containerName="sequence-graph-12f5e"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921571    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kserve-container"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921576    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c5545c-58dd-4a4a-b965-cfbb88ac7ba0" containerName="kube-rbac-proxy"
Jul 28 16:55:16.921607 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.921583    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4cf707-e78a-4ca5-ab2c-09f0388d2f22" containerName="splitter-graph-2717b"
Jul 28 16:55:16.925521 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.925501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:16.927653 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.927629    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b9a8c-serving-cert\""
Jul 28 16:55:16.927753 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.927633    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b9a8c-kube-rbac-proxy-sar-config\""
Jul 28 16:55:16.932230 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:16.932203    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 16:55:17.015781 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.015747    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.015781 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.015781    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.116700 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.116664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.116700 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.116703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.117358 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.117332    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.119075 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.119055    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls\") pod \"switch-graph-b9a8c-77584696f7-crzm9\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") " pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.236313 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.236241    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:17.356510 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:17.356479    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 16:55:17.359574 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:55:17.359549    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c15256_d99c_4357_8f32_3a89d6d228e0.slice/crio-d7145858e22430e9081ee2dc8754a354547e319c2e39178e3353dc1b49c293b8 WatchSource:0}: Error finding container d7145858e22430e9081ee2dc8754a354547e319c2e39178e3353dc1b49c293b8: Status 404 returned error can't find the container with id d7145858e22430e9081ee2dc8754a354547e319c2e39178e3353dc1b49c293b8
Jul 28 16:55:18.034096 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:18.034063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" event={"ID":"49c15256-d99c-4357-8f32-3a89d6d228e0","Type":"ContainerStarted","Data":"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"}
Jul 28 16:55:18.034462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:18.034112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" event={"ID":"49c15256-d99c-4357-8f32-3a89d6d228e0","Type":"ContainerStarted","Data":"d7145858e22430e9081ee2dc8754a354547e319c2e39178e3353dc1b49c293b8"}
Jul 28 16:55:18.034462 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:18.034245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:18.055848 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:18.055808    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podStartSLOduration=2.055794978 podStartE2EDuration="2.055794978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:55:18.054706856 +0000 UTC m=+2032.787336535" watchObservedRunningTime="2026-07-28 16:55:18.055794978 +0000 UTC m=+2032.788424655"
Jul 28 16:55:24.043131 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:24.043103    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 16:55:27.341509 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.341480    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 16:55:27.344572 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.344557    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.346826 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.346805    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b6e34-kube-rbac-proxy-sar-config\""
Jul 28 16:55:27.347113 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.347095    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b6e34-serving-cert\""
Jul 28 16:55:27.354735 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.354712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 16:55:27.385549 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.385527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.385663 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.385569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.486616 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.486586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.486825 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.486633    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.486825 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:55:27.486720    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-b6e34-serving-cert: secret "splitter-graph-b6e34-serving-cert" not found
Jul 28 16:55:27.486825 ip-10-0-139-46 kubenswrapper[2555]: E0728 16:55:27.486779    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls podName:bd61f92a-f1d3-444a-b990-747317d94b74 nodeName:}" failed. No retries permitted until 2026-07-28 16:55:27.98676427 +0000 UTC m=+2042.719393925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls") pod "splitter-graph-b6e34-54964b94f7-q8lxq" (UID: "bd61f92a-f1d3-444a-b990-747317d94b74") : secret "splitter-graph-b6e34-serving-cert" not found
Jul 28 16:55:27.487205 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.487190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.991900 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.991870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:27.994302 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:27.994275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") pod \"splitter-graph-b6e34-54964b94f7-q8lxq\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") " pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:28.254326 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:28.254261    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:28.373409 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:28.373378    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 16:55:28.377698 ip-10-0-139-46 kubenswrapper[2555]: W0728 16:55:28.377661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd61f92a_f1d3_444a_b990_747317d94b74.slice/crio-8fe337abddcbc2c63500ae0dc8d5a18d164d1e62fbabeed1c4e3f741ee0bca64 WatchSource:0}: Error finding container 8fe337abddcbc2c63500ae0dc8d5a18d164d1e62fbabeed1c4e3f741ee0bca64: Status 404 returned error can't find the container with id 8fe337abddcbc2c63500ae0dc8d5a18d164d1e62fbabeed1c4e3f741ee0bca64
Jul 28 16:55:29.078741 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:29.078700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" event={"ID":"bd61f92a-f1d3-444a-b990-747317d94b74","Type":"ContainerStarted","Data":"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"}
Jul 28 16:55:29.078903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:29.078752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" event={"ID":"bd61f92a-f1d3-444a-b990-747317d94b74","Type":"ContainerStarted","Data":"8fe337abddcbc2c63500ae0dc8d5a18d164d1e62fbabeed1c4e3f741ee0bca64"}
Jul 28 16:55:29.078903 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:29.078799    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:55:29.093690 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:29.093643    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podStartSLOduration=2.09362937 podStartE2EDuration="2.09362937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:55:29.093090699 +0000 UTC m=+2043.825720379" watchObservedRunningTime="2026-07-28 16:55:29.09362937 +0000 UTC m=+2043.826259047"
Jul 28 16:55:35.087129 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:55:35.087103    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 16:56:25.745891 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:56:25.745818    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 16:56:25.751962 ip-10-0-139-46 kubenswrapper[2555]: I0728 16:56:25.751945    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:01:25.766323 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:01:25.766291    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:01:25.772834 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:01:25.772815    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:03:41.965173 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:41.965130    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 17:03:41.965768 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:41.965339    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" containerID="cri-o://f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08" gracePeriod=30
Jul 28 17:03:42.111038 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:42.110983    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 17:03:42.111268 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:42.111233    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" containerID="cri-o://1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856" gracePeriod=30
Jul 28 17:03:42.111414 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:42.111280    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kube-rbac-proxy" containerID="cri-o://67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8" gracePeriod=30
Jul 28 17:03:42.796241 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:42.796207    2555 generic.go:356] "Generic (PLEG): container finished" podID="689c2464-57c6-4a45-8ab0-4de108999c77" containerID="67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8" exitCode=2
Jul 28 17:03:42.796241 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:42.796247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerDied","Data":"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"}
Jul 28 17:03:44.861884 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:44.861828    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 17:03:44.867125 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:44.867092    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:03:44.960855 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:44.960829    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 17:03:45.007299 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.007221    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj56j\" (UniqueName: \"kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j\") pod \"689c2464-57c6-4a45-8ab0-4de108999c77\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") "
Jul 28 17:03:45.007299 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.007261    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config\") pod \"689c2464-57c6-4a45-8ab0-4de108999c77\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") "
Jul 28 17:03:45.007484 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.007305    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls\") pod \"689c2464-57c6-4a45-8ab0-4de108999c77\" (UID: \"689c2464-57c6-4a45-8ab0-4de108999c77\") "
Jul 28 17:03:45.007609 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.007590    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "689c2464-57c6-4a45-8ab0-4de108999c77" (UID: "689c2464-57c6-4a45-8ab0-4de108999c77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:03:45.009501 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.009484    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j" (OuterVolumeSpecName: "kube-api-access-rj56j") pod "689c2464-57c6-4a45-8ab0-4de108999c77" (UID: "689c2464-57c6-4a45-8ab0-4de108999c77"). InnerVolumeSpecName "kube-api-access-rj56j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:03:45.009568 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.009539    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "689c2464-57c6-4a45-8ab0-4de108999c77" (UID: "689c2464-57c6-4a45-8ab0-4de108999c77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:03:45.086675 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.086638    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:03:45.108503 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.108476    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c2464-57c6-4a45-8ab0-4de108999c77-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:03:45.113759 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.108502    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj56j\" (UniqueName: \"kubernetes.io/projected/689c2464-57c6-4a45-8ab0-4de108999c77-kube-api-access-rj56j\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:03:45.113759 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.108866    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c2464-57c6-4a45-8ab0-4de108999c77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:03:45.808428 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.808392    2555 generic.go:356] "Generic (PLEG): container finished" podID="689c2464-57c6-4a45-8ab0-4de108999c77" containerID="1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856" exitCode=0
Jul 28 17:03:45.808629 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.808455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerDied","Data":"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"}
Jul 28 17:03:45.808629 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.808466    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"
Jul 28 17:03:45.808629 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.808536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j" event={"ID":"689c2464-57c6-4a45-8ab0-4de108999c77","Type":"ContainerDied","Data":"5c14377325fca4a1611691b442db71db8abe2eab8f600fedcfd8c4a3a43568e6"}
Jul 28 17:03:45.808629 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.808563    2555 scope.go:117] "RemoveContainer" containerID="67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"
Jul 28 17:03:45.816984 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.816965    2555 scope.go:117] "RemoveContainer" containerID="1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"
Jul 28 17:03:45.824732 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.824710    2555 scope.go:117] "RemoveContainer" containerID="67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"
Jul 28 17:03:45.825024 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:03:45.824982    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8\": container with ID starting with 67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8 not found: ID does not exist" containerID="67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"
Jul 28 17:03:45.825093 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.825038    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8"} err="failed to get container status \"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8\": rpc error: code = NotFound desc = could not find container \"67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8\": container with ID starting with 67f26fd7567bec126116fc8b236a7ffc67c17c702cd9af6c30f1d273f05866e8 not found: ID does not exist"
Jul 28 17:03:45.825093 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.825062    2555 scope.go:117] "RemoveContainer" containerID="1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"
Jul 28 17:03:45.825345 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:03:45.825329    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856\": container with ID starting with 1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856 not found: ID does not exist" containerID="1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"
Jul 28 17:03:45.825402 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.825349    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856"} err="failed to get container status \"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856\": rpc error: code = NotFound desc = could not find container \"1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856\": container with ID starting with 1a5f7e7d731c8762f7d637f9bc6d66c3ee99a42582f3757def044dd4169bb856 not found: ID does not exist"
Jul 28 17:03:45.829297 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.829276    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 17:03:45.833126 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:45.833107    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6e34-predictor-d4bb678f4-x429j"]
Jul 28 17:03:47.823083 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:47.823051    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" path="/var/lib/kubelet/pods/689c2464-57c6-4a45-8ab0-4de108999c77/volumes"
Jul 28 17:03:50.086331 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:50.086287    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:03:55.086788 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:55.086750    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:03:55.087195 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:03:55.086862    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 17:04:00.086333 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:00.086292    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:04:05.086120 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:05.086081    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:04:10.086082 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:10.086041    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:04:12.108797 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.108772    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 17:04:12.194234 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.194203    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") pod \"bd61f92a-f1d3-444a-b990-747317d94b74\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") "
Jul 28 17:04:12.194413 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.194272    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle\") pod \"bd61f92a-f1d3-444a-b990-747317d94b74\" (UID: \"bd61f92a-f1d3-444a-b990-747317d94b74\") "
Jul 28 17:04:12.194666 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.194634    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bd61f92a-f1d3-444a-b990-747317d94b74" (UID: "bd61f92a-f1d3-444a-b990-747317d94b74"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:04:12.196574 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.196543    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd61f92a-f1d3-444a-b990-747317d94b74" (UID: "bd61f92a-f1d3-444a-b990-747317d94b74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:04:12.295371 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.295333    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd61f92a-f1d3-444a-b990-747317d94b74-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:04:12.295371 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.295364    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd61f92a-f1d3-444a-b990-747317d94b74-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:04:12.904562 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.904529    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd61f92a-f1d3-444a-b990-747317d94b74" containerID="f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08" exitCode=0
Jul 28 17:04:12.904746 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.904573    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"
Jul 28 17:04:12.904746 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.904589    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" event={"ID":"bd61f92a-f1d3-444a-b990-747317d94b74","Type":"ContainerDied","Data":"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"}
Jul 28 17:04:12.904746 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.904627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq" event={"ID":"bd61f92a-f1d3-444a-b990-747317d94b74","Type":"ContainerDied","Data":"8fe337abddcbc2c63500ae0dc8d5a18d164d1e62fbabeed1c4e3f741ee0bca64"}
Jul 28 17:04:12.904746 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.904648    2555 scope.go:117] "RemoveContainer" containerID="f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"
Jul 28 17:04:12.913370 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.913351    2555 scope.go:117] "RemoveContainer" containerID="f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"
Jul 28 17:04:12.913699 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:04:12.913680    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08\": container with ID starting with f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08 not found: ID does not exist" containerID="f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"
Jul 28 17:04:12.913761 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.913709    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08"} err="failed to get container status \"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08\": rpc error: code = NotFound desc = could not find container \"f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08\": container with ID starting with f742461907127a1b2b9e3484d59ae651b812510715a68f6280d7d69f540a6c08 not found: ID does not exist"
Jul 28 17:04:12.925618 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.925591    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 17:04:12.927970 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:12.927945    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b6e34-54964b94f7-q8lxq"]
Jul 28 17:04:13.823242 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:04:13.823205    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" path="/var/lib/kubelet/pods/bd61f92a-f1d3-444a-b990-747317d94b74/volumes"
Jul 28 17:06:25.786647 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:06:25.786531    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:06:25.793269 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:06:25.793251    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:11:25.807296 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:25.807184    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:11:25.814273 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:25.814251    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:11:36.187612 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:36.187575    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 17:11:36.188136 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:36.187791    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" containerID="cri-o://1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435" gracePeriod=30
Jul 28 17:11:36.389372 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:36.389339    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 17:11:36.389633 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:36.389571    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container" containerID="cri-o://18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014" gracePeriod=30
Jul 28 17:11:36.389782 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:36.389632    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kube-rbac-proxy" containerID="cri-o://bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a" gracePeriod=30
Jul 28 17:11:37.471715 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:37.471681    2555 generic.go:356] "Generic (PLEG): container finished" podID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerID="bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a" exitCode=2
Jul 28 17:11:37.472120 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:37.471755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerDied","Data":"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"}
Jul 28 17:11:39.042111 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.042064    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:11:39.632238 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.632213    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 17:11:39.671811 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.671778    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") pod \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") "
Jul 28 17:11:39.671983 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.671830    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config\") pod \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") "
Jul 28 17:11:39.671983 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.671849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6zjd\" (UniqueName: \"kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd\") pod \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\" (UID: \"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2\") "
Jul 28 17:11:39.672244 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.672217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" (UID: "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:11:39.673909 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.673872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" (UID: "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:11:39.674043 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.673932    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd" (OuterVolumeSpecName: "kube-api-access-p6zjd") pod "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" (UID: "e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2"). InnerVolumeSpecName "kube-api-access-p6zjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:11:39.772272 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.772240    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:11:39.772272 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.772270    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zjd\" (UniqueName: \"kubernetes.io/projected/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-kube-api-access-p6zjd\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:11:39.772476 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:39.772281    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:11:40.484294 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.484258    2555 generic.go:356] "Generic (PLEG): container finished" podID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerID="18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014" exitCode=0
Jul 28 17:11:40.484704 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.484318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"
Jul 28 17:11:40.484704 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.484343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerDied","Data":"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"}
Jul 28 17:11:40.484704 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.484400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8" event={"ID":"e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2","Type":"ContainerDied","Data":"e5a393d974e9d056654dcb444be057427e8edc9ba62278eb74fc2cd46891811f"}
Jul 28 17:11:40.484704 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.484425    2555 scope.go:117] "RemoveContainer" containerID="bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"
Jul 28 17:11:40.492653 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.492638    2555 scope.go:117] "RemoveContainer" containerID="18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"
Jul 28 17:11:40.500172 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.500151    2555 scope.go:117] "RemoveContainer" containerID="bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"
Jul 28 17:11:40.500249 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.500159    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 17:11:40.500439 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:11:40.500423    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a\": container with ID starting with bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a not found: ID does not exist" containerID="bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"
Jul 28 17:11:40.500480 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.500447    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a"} err="failed to get container status \"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a\": rpc error: code = NotFound desc = could not find container \"bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a\": container with ID starting with bb42ccac7f63d549ada8c6c084350ff772d7b08a1f6160c93fdc48ead7d21d2a not found: ID does not exist"
Jul 28 17:11:40.500480 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.500466    2555 scope.go:117] "RemoveContainer" containerID="18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"
Jul 28 17:11:40.500715 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:11:40.500692    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014\": container with ID starting with 18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014 not found: ID does not exist" containerID="18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"
Jul 28 17:11:40.500775 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.500725    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014"} err="failed to get container status \"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014\": rpc error: code = NotFound desc = could not find container \"18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014\": container with ID starting with 18e812b76ed3c29055e943ca446d3f195fe11885759669b86fc70d2045d86014 not found: ID does not exist"
Jul 28 17:11:40.503480 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:40.503461    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9a8c-predictor-57b5cbbd5b-cm5g8"]
Jul 28 17:11:41.823285 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:41.823253    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" path="/var/lib/kubelet/pods/e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2/volumes"
Jul 28 17:11:44.042113 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:44.042067    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:11:49.041711 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:49.041672    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:11:49.042140 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:49.041771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 17:11:54.042033 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:54.041976    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:11:54.069907 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:54.069869    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:54.954914 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:54.954885    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:55.815757 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:55.815728    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:56.651717 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:56.651682    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:57.447581 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:57.447556    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:58.303374 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:58.303345    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:11:59.041902 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:59.041856    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:11:59.214470 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:11:59.214440    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:00.123635 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:00.123601    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:01.015220 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:01.015184    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:01.920940 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:01.920911    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:02.822990 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:02.822948    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:03.659490 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:03.659462    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:04.041961 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:04.041927    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:12:04.511973 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:04.511945    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:05.381033 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:05.380993    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b9a8c-77584696f7-crzm9_49c15256-d99c-4357-8f32-3a89d6d228e0/switch-graph-b9a8c/0.log"
Jul 28 17:12:06.341031 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.340986    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 17:12:06.359672 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.359612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls\") pod \"49c15256-d99c-4357-8f32-3a89d6d228e0\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") "
Jul 28 17:12:06.359913 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.359709    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle\") pod \"49c15256-d99c-4357-8f32-3a89d6d228e0\" (UID: \"49c15256-d99c-4357-8f32-3a89d6d228e0\") "
Jul 28 17:12:06.360185 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.360156    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "49c15256-d99c-4357-8f32-3a89d6d228e0" (UID: "49c15256-d99c-4357-8f32-3a89d6d228e0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:06.362212 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.362187    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49c15256-d99c-4357-8f32-3a89d6d228e0" (UID: "49c15256-d99c-4357-8f32-3a89d6d228e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:06.460751 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.460656    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c15256-d99c-4357-8f32-3a89d6d228e0-proxy-tls\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:12:06.460751 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.460690    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49c15256-d99c-4357-8f32-3a89d6d228e0-openshift-service-ca-bundle\") on node \"ip-10-0-139-46.ec2.internal\" DevicePath \"\""
Jul 28 17:12:06.577623 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.577583    2555 generic.go:356] "Generic (PLEG): container finished" podID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerID="1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435" exitCode=0
Jul 28 17:12:06.577828 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.577632    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"
Jul 28 17:12:06.577828 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.577649    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" event={"ID":"49c15256-d99c-4357-8f32-3a89d6d228e0","Type":"ContainerDied","Data":"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"}
Jul 28 17:12:06.577828 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.577697    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9" event={"ID":"49c15256-d99c-4357-8f32-3a89d6d228e0","Type":"ContainerDied","Data":"d7145858e22430e9081ee2dc8754a354547e319c2e39178e3353dc1b49c293b8"}
Jul 28 17:12:06.577828 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.577722    2555 scope.go:117] "RemoveContainer" containerID="1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"
Jul 28 17:12:06.586254 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.586236    2555 scope.go:117] "RemoveContainer" containerID="1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"
Jul 28 17:12:06.586534 ip-10-0-139-46 kubenswrapper[2555]: E0728 17:12:06.586513    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435\": container with ID starting with 1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435 not found: ID does not exist" containerID="1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"
Jul 28 17:12:06.586580 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.586545    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435"} err="failed to get container status \"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435\": rpc error: code = NotFound desc = could not find container \"1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435\": container with ID starting with 1971d01f88f0e92bd6a85607a1acd53769d9371a4ca66f974ab58971c4f90435 not found: ID does not exist"
Jul 28 17:12:06.598932 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.598906    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 17:12:06.603974 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:06.603952    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b9a8c-77584696f7-crzm9"]
Jul 28 17:12:07.594571 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594534    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-464v8/must-gather-hh5jx"]
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594785    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594796    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594804    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kube-rbac-proxy"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594812    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kube-rbac-proxy"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594821    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594827    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594841    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594846    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594852    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594858    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594866    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kube-rbac-proxy"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kube-rbac-proxy"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594905    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" containerName="switch-graph-b9a8c"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594912    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594917    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85b47f1-f9d8-46f3-8d95-ae9e1c2997b2" containerName="kube-rbac-proxy"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd61f92a-f1d3-444a-b990-747317d94b74" containerName="splitter-graph-b6e34"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594929    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kserve-container"
Jul 28 17:12:07.594954 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.594933    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="689c2464-57c6-4a45-8ab0-4de108999c77" containerName="kube-rbac-proxy"
Jul 28 17:12:07.598827 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.598810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.601183 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.601157    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-464v8\"/\"kube-root-ca.crt\""
Jul 28 17:12:07.601317 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.601278    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-464v8\"/\"default-dockercfg-mdmkx\""
Jul 28 17:12:07.601420 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.601406    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-464v8\"/\"openshift-service-ca.crt\""
Jul 28 17:12:07.608184 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.608163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-464v8/must-gather-hh5jx"]
Jul 28 17:12:07.669268 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.669229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csrl\" (UniqueName: \"kubernetes.io/projected/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-kube-api-access-4csrl\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.669458 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.669290    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-must-gather-output\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.770274 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.770231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-must-gather-output\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.770465 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.770298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csrl\" (UniqueName: \"kubernetes.io/projected/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-kube-api-access-4csrl\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.770600 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.770578    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-must-gather-output\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.779769 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.779732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csrl\" (UniqueName: \"kubernetes.io/projected/1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9-kube-api-access-4csrl\") pod \"must-gather-hh5jx\" (UID: \"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9\") " pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:07.822899 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.822855    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c15256-d99c-4357-8f32-3a89d6d228e0" path="/var/lib/kubelet/pods/49c15256-d99c-4357-8f32-3a89d6d228e0/volumes"
Jul 28 17:12:07.908202 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:07.908113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-464v8/must-gather-hh5jx"
Jul 28 17:12:08.035892 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:08.035856    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-464v8/must-gather-hh5jx"]
Jul 28 17:12:08.039902 ip-10-0-139-46 kubenswrapper[2555]: W0728 17:12:08.039870    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4bda45_e417_4cc0_b6f7_4d324ff4ebe9.slice/crio-20a2d89bd6f143c0aa825b7f74654d5e8755bd782ea11f044f90d220f1097b99 WatchSource:0}: Error finding container 20a2d89bd6f143c0aa825b7f74654d5e8755bd782ea11f044f90d220f1097b99: Status 404 returned error can't find the container with id 20a2d89bd6f143c0aa825b7f74654d5e8755bd782ea11f044f90d220f1097b99
Jul 28 17:12:08.041728 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:08.041716    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:12:08.587462 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:08.587420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-464v8/must-gather-hh5jx" event={"ID":"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9","Type":"ContainerStarted","Data":"20a2d89bd6f143c0aa825b7f74654d5e8755bd782ea11f044f90d220f1097b99"}
Jul 28 17:12:09.593574 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:09.593538    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-464v8/must-gather-hh5jx" event={"ID":"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9","Type":"ContainerStarted","Data":"715b149662f28d0296ec621f41176941668c26cefea29f548b412c6b19df2ee7"}
Jul 28 17:12:09.594079 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:09.593588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-464v8/must-gather-hh5jx" event={"ID":"1e4bda45-e417-4cc0-b6f7-4d324ff4ebe9","Type":"ContainerStarted","Data":"31ccf8f918dd9b668945fc86399a558c6ab72939d5bd1e57c92e75a6a4aa9986"}
Jul 28 17:12:09.615935 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:09.615874    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-464v8/must-gather-hh5jx" podStartSLOduration=1.9296304499999999 podStartE2EDuration="2.615857121s" totalImagesPullingTime="686.226671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:09.61495082 +0000 UTC m=+3044.347580497" watchObservedRunningTime="2026-07-28 17:12:09.615857121 +0000 UTC m=+3044.348486798"
Jul 28 17:12:10.261641 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:10.261608    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2nvj5_5bf00c4a-8cb3-48f4-a3d4-60e55b40d205/global-pull-secret-syncer/0.log"
Jul 28 17:12:10.433946 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:10.433907    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9tqnm_0686c156-0f69-48e5-91e9-18b32c83249e/konnectivity-agent/0.log"
Jul 28 17:12:10.492285 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:10.492247    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-46.ec2.internal_39f953389fbb7b355964e9b9fe5538ef/haproxy/0.log"
Jul 28 17:12:14.656761 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:14.656722    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-22tpm_bab389b2-348d-4ff4-8686-150bbf487b9f/node-exporter/0.log"
Jul 28 17:12:14.680520 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:14.680454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-22tpm_bab389b2-348d-4ff4-8686-150bbf487b9f/kube-rbac-proxy/0.log"
Jul 28 17:12:14.704873 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:14.704829    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-22tpm_bab389b2-348d-4ff4-8686-150bbf487b9f/init-textfile/0.log"
Jul 28 17:12:16.894758 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:16.894726    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-r849p_e8c8383e-2bb0-435c-abf2-59869f817269/networking-console-plugin/0.log"
Jul 28 17:12:18.424861 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.424816    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"]
Jul 28 17:12:18.430799 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.430769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.434780 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.434749    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"]
Jul 28 17:12:18.570967 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.570934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-sys\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.571177 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.570978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-podres\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.571177 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.571039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-proc\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.571177 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.571101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjcg\" (UniqueName: \"kubernetes.io/projected/99915d8a-5292-4b03-a457-b6026a72e800-kube-api-access-bdjcg\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.571177 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.571153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-lib-modules\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.672538 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.672496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjcg\" (UniqueName: \"kubernetes.io/projected/99915d8a-5292-4b03-a457-b6026a72e800-kube-api-access-bdjcg\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.672818 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.672795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-lib-modules\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.672973 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.672956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-sys\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673142 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.673109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-sys\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673313 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.673132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-podres\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673499 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.673463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-proc\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673691 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.672993    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-lib-modules\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673793 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.673228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-podres\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.673793 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.673762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99915d8a-5292-4b03-a457-b6026a72e800-proc\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.682298 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.681423    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjcg\" (UniqueName: \"kubernetes.io/projected/99915d8a-5292-4b03-a457-b6026a72e800-kube-api-access-bdjcg\") pod \"perf-node-gather-daemonset-7ds84\" (UID: \"99915d8a-5292-4b03-a457-b6026a72e800\") " pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.743489 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.743454    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:18.909311 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:18.909219    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"]
Jul 28 17:12:19.456551 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.456518    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w65fs_65c2aac8-86e5-45ac-aa53-7877cf0b3bb0/dns/0.log"
Jul 28 17:12:19.483973 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.483930    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w65fs_65c2aac8-86e5-45ac-aa53-7877cf0b3bb0/kube-rbac-proxy/0.log"
Jul 28 17:12:19.566418 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.566385    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7q76r_a80d9dc6-f175-44da-be32-99667ffe2331/dns-node-resolver/0.log"
Jul 28 17:12:19.646744 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.646706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84" event={"ID":"99915d8a-5292-4b03-a457-b6026a72e800","Type":"ContainerStarted","Data":"d97298e5e56835deab273761694d48c3b34bc822a3fe4caf442a4f374481485c"}
Jul 28 17:12:19.647172 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.647143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84" event={"ID":"99915d8a-5292-4b03-a457-b6026a72e800","Type":"ContainerStarted","Data":"ba1a0c160a095974c952cc03ea86a36d954da078aa81a06b8d564d5e7e6283dc"}
Jul 28 17:12:19.647346 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.647331    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:19.665430 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:19.665372    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84" podStartSLOduration=1.665352474 podStartE2EDuration="1.665352474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:19.664163445 +0000 UTC m=+3054.396793121" watchObservedRunningTime="2026-07-28 17:12:19.665352474 +0000 UTC m=+3054.397982153"
Jul 28 17:12:20.181292 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:20.181268    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p5lv9_32d5b350-ecea-466c-a4a2-198e0e8ee180/node-ca/0.log"
Jul 28 17:12:21.239689 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:21.239661    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9s8s_e0834160-3b3d-49de-b396-5914cb537776/serve-healthcheck-canary/0.log"
Jul 28 17:12:21.824013 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:21.823981    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxrnk_fd850d07-0f15-4189-a81e-46195dc699da/kube-rbac-proxy/0.log"
Jul 28 17:12:21.845625 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:21.845599    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxrnk_fd850d07-0f15-4189-a81e-46195dc699da/exporter/0.log"
Jul 28 17:12:21.868164 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:21.868126    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxrnk_fd850d07-0f15-4189-a81e-46195dc699da/extractor/0.log"
Jul 28 17:12:24.050321 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:24.050285    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85d8b8ddfc-f9w9s_3246ae98-96ac-4ed4-b38c-11450beb213e/manager/0.log"
Jul 28 17:12:24.213876 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:24.213827    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-5mq4z_1af2fc51-9647-48df-864f-252b55ef86b9/manager/0.log"
Jul 28 17:12:25.662705 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:25.662675    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-464v8/perf-node-gather-daemonset-7ds84"
Jul 28 17:12:28.426213 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:28.426124    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-2h8xl_eb780086-a4dc-4130-b43b-2b50e3b8ff01/migrator/0.log"
Jul 28 17:12:28.452290 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:28.452264    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-2h8xl_eb780086-a4dc-4130-b43b-2b50e3b8ff01/graceful-termination/0.log"
Jul 28 17:12:29.928781 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:29.928758    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/kube-multus-additional-cni-plugins/0.log"
Jul 28 17:12:29.949619 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:29.949594    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/egress-router-binary-copy/0.log"
Jul 28 17:12:29.969499 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:29.969469    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/cni-plugins/0.log"
Jul 28 17:12:29.990044 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:29.990014    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/bond-cni-plugin/0.log"
Jul 28 17:12:30.010895 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.010864    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/routeoverride-cni/0.log"
Jul 28 17:12:30.031946 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.031919    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/whereabouts-cni-bincopy/0.log"
Jul 28 17:12:30.051687 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.051661    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2xw8_39385f3c-509b-4451-8d3c-7cd58f3599bd/whereabouts-cni/0.log"
Jul 28 17:12:30.262523 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.262492    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gfcgw_608e9c4b-1e1e-4f48-9513-7d2770221dd6/kube-multus/0.log"
Jul 28 17:12:30.351770 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.351734    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ljfnz_096be67a-895c-4ebb-a921-30f37159c2dc/network-metrics-daemon/0.log"
Jul 28 17:12:30.371419 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:30.371394    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ljfnz_096be67a-895c-4ebb-a921-30f37159c2dc/kube-rbac-proxy/0.log"
Jul 28 17:12:31.128041 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.127993    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-controller/0.log"
Jul 28 17:12:31.144527 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.144501    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/1.log"
Jul 28 17:12:31.144697 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.144511    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovn-acl-logging/0.log"
Jul 28 17:12:31.162561 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.162532    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/kube-rbac-proxy-node/0.log"
Jul 28 17:12:31.182080 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.182053    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 17:12:31.203698 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.203675    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/northd/0.log"
Jul 28 17:12:31.224132 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.224107    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/nbdb/0.log"
Jul 28 17:12:31.244279 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.244256    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/sbdb/0.log"
Jul 28 17:12:31.289017 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:31.288975    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9vmh_3c5d987e-a8d3-4f77-949f-650857f72a9b/ovnkube-controller/0.log"
Jul 28 17:12:32.642203 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:32.642156    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-457ph_9f93020b-f7be-45c6-af6a-598acab297da/check-endpoints/0.log"
Jul 28 17:12:32.715407 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:32.715374    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tgggh_f516c902-f1cc-4df1-bf69-53b94ea9d874/network-check-target-container/0.log"
Jul 28 17:12:33.565180 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:33.565149    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4br26_5c840a7d-6bfb-423e-a6ce-696c9b2a3ef7/iptables-alerter/0.log"
Jul 28 17:12:34.217738 ip-10-0-139-46 kubenswrapper[2555]: I0728 17:12:34.217695    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-g824h_e2c19cdc-09b8-450f-b38d-e34b65687d9f/tuned/0.log"