Sep 17 18:27:11.373476 ip-10-0-143-176 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:27:11.373486 ip-10-0-143-176 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:27:11.373496 ip-10-0-143-176 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:27:11.373676 ip-10-0-143-176 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:27:21.509730 ip-10-0-143-176 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:27:21.509744 ip-10-0-143-176 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9dd97197e2d64cfb835e9cd1e2ce66ed --
Sep 17 18:29:17.328171 ip-10-0-143-176 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:17.723124 ip-10-0-143-176 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:17.723124 ip-10-0-143-176 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:17.723124 ip-10-0-143-176 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:17.723124 ip-10-0-143-176 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:17.723124 ip-10-0-143-176 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:17.725103 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.725018    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:17.728505 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728491    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728507    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728511    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728514    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728517    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728520    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728523    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728526    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728530    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728535    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728538    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728541    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728544    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728547    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728550    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728552    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.728548 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728555    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728558    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728561    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728563    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728566    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728573    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728576    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728579    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728581    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728584    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728586    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728589    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728591    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728594    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728596    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728599    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728601    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728604    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728606    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728608    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.728948 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728611    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728613    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728616    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728618    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728621    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728623    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728625    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728628    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728631    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728633    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728636    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728638    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728640    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728643    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728647    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728649    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728652    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728654    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728660    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728663    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.729467 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728666    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728668    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728671    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728673    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728676    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728678    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728680    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728683    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728685    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728688    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728690    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728693    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728695    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728698    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728700    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728704    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728708    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728710    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728715    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728718    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.729972 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728721    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728725    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728727    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728730    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728732    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728735    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728738    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728741    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728743    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728746    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728749    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728753    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728756    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728758    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728761    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728764    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728766    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728770    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728773    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.730453 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728776    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728779    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728782    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.728785    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729537    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729544    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729547    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729550    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729554    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729558    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729561    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729565    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729567    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729570    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729573    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729575    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729578    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729580    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729584    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729587    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.730935 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729589    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729592    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729594    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729597    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729600    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729603    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729605    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729608    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729611    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729613    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729616    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729618    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729621    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729623    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729626    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729628    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729631    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729633    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729635    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729638    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.731428 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729641    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729643    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729646    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729648    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729651    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729653    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729656    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729661    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729664    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729667    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729671    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729674    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729676    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729679    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729682    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729685    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729687    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729690    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729693    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729696    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.731963 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729698    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729701    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729704    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729707    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729709    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729712    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729714    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729717    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729719    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729722    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729724    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729727    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729729    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729732    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729734    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729737    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729739    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729742    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729744    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729747    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.732461 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729749    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729752    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729755    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729758    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729760    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729763    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729766    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729768    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729772    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729776    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729779    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729782    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729785    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729788    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729790    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729793    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729795    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729797    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729800    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.732973 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729802    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729805    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729807    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.729810    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730201    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730207    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730210    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730213    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730216    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730220    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730224    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730227    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730230    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730233    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730236    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730242    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730244    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730247    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730250    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730253    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.733437 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730256    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730259    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730262    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730265    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730269    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730271    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730274    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730277    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730279    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730282    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730285    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730287    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730290    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730292    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730295    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730298    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730300    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730303    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730305    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730308    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.733934 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730311    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730314    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730317    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730319    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730322    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730326    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730329    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730334    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730336    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730339    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730341    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730344    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730347    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730349    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730352    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730355    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730357    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730360    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730363    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.734412 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730365    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730368    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730370    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730373    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730375    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730378    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730380    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730383    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730385    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730388    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730391    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730394    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730396    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730399    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730401    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730404    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730406    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730409    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730411    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730414    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.734890 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730418    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730420    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730423    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730425    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730428    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730431    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730433    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730436    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730438    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730441    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730443    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730446    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730450    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730454    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730457    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730460    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730463    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730466    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730469    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.735381 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730472    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730474    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730477    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730480    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.730483    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730565    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730611    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730618    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730622    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730626    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730629    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730635    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730640    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730644    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730647    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730651    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730654    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730657    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730660    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730663    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730666    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730669    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730672    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730675    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:17.735862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730678    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730682    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730685    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730688    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730692    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730695    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730698    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730701    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730704    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730707    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730709    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730712    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730715    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730719    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730722    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730724    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730727    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730730    2559 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730733    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730738    2559 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730741    2559 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730745    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730748    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730751    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:17.736446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730755    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730758    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730761    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730764    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730767    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730770    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730773    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730775    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730778    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730781    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730784    2559 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730788    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730791    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730794    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730797    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730800    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730803    2559 flags.go:64] FLAG: --help="false"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730812    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-143-176.ec2.internal"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730816    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730819    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730822    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730837    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730841    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730844    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:17.737045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730847    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730850    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730853    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730857    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730861    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730864    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730867    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730870    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730873    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730876    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730879    2559 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730882    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730885    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730888    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730893    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730896    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730899    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730902    2559 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730905    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730909    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730912    2559 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730915    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730919    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730922    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730926    2559 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:17.737737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730929    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730932    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730935    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730938    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730941    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730944    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730948    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730955    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730958    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730961    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730966    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730969    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730974    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730977    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730981    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730983    2559 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730986    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730989    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00778758417e3f258"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730992    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730995    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.730998    2559 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731001    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731005    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731007    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:17.738677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731010    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731013    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731016    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731020    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731023    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731026    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731028    2559 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731031    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731034    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731037    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731040    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731043    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731046    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731049    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731052    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731054    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731057    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731060    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731064    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731067    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731070    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731074    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731079    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731082    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731085    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:17.739474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731089    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731091    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731094    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731097    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731100    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731103    2559 flags.go:64] FLAG: --v="2"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731107    2559 flags.go:64] FLAG: --version="false"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731110    2559 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731114    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731118    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731234    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731238    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731241    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731243    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731247    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731250    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731253    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731255    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731258    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731261    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731264    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.740093 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731266    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731270    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731273    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731276    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731280    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731283    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731285    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731289    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731292    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731295    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731298    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731300    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731303    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731306    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731308    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731311    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731314    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731316    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731319    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.740603 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731322    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731325    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731327    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731330    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731333    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731336    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731339    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731341    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731344    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731347    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731350    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731352    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731355    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731358    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731360    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731363    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731365    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731369    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731372    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.741107 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731376    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731381    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731384    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731387    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731390    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731393    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731395    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731398    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731402    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731404    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731407    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731410    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731413    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731416    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731419    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731422    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731424    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731427    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731432    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731436    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.741601 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731439    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731442    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731444    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731447    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731450    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731453    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731456    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731459    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731461    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731464    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731468    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731471    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731473    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731477    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731480    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731483    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731485    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731488    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731491    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731494    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.742109 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731496    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731499    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731501    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731504    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731506    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731509    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731512    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731514    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731517    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.731520    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.731959    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.740004    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.740019    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.740884    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.740891    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:17.742599 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740948    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740953    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740957    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740960    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740963    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740965    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740968    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740971    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740974    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740977    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740979    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740982    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740985    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740987    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740990    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740993    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740996    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.740999    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741002    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741005    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.743003 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741009    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741012    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741015    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741017    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741020    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741023    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741026    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741029    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741031    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741034    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741037    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741039    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741042    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741045    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741047    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741050    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741052    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741055    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741058    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.743496 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741062    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741066    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741069    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741071    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741074    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741077    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741080    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741083    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741086    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741089    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741092    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741095    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741099    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741103    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741106    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741109    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741111    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741114    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741117    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.744035 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741121    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741124    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741127    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741131    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741133    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741136    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741139    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741142    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741144    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741147    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741150    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741153    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741155    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741158    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741161    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741164    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741166    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741169    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741171    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741174    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.744501 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741176    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741180    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741183    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741186    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741189    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741191    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741194    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741197    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741199    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741202    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741204    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741207    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741209    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741212    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741215    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741218    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741221    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741224    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741226    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741229    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.745033 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741231    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.741236    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741350    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741355    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741358    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741361    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741364    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741367    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741370    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741372    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741375    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741377    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741380    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741383    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741390    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:17.745579 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741393    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741396    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741399    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741402    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741405    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741407    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741410    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741413    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741415    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741418    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741420    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741423    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741426    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741428    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741431    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741434    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741436    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741439    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741441    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741444    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741447    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:17.745962 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741449    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741452    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741454    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741457    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741460    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741462    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741465    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741467    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741470    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741472    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741476    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741479    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741481    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741484    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741487    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741489    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741492    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741495    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741497    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:17.746470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741500    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741502    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741505    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741507    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741510    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741513    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741516    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741518    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741521    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741523    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741526    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741528    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741531    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741533    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741536    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741538    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741541    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741543    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741546    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741548    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:17.746991 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741551    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741554    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741557    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741559    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741562    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741565    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741568    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741572    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741575    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741578    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741582    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741585    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741587    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741591    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741594    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741597    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741600    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741603    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741606    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:17.747477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741610    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741613    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741616    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741619    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741622    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741624    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:17.741627    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.741632    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.741745    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.744947    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.745908    2559 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.746007    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:17.748038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.746037    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:17.748896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.748884    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:17.750015 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.749977    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:17.758455 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.758438    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:17.759990 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.759975    2559 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:17.760784 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.760758    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:17.766661 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.766636    2559 fs.go:135] Filesystem UUIDs: map[669944fc-9428-4842-9a41-62f6719b6d7a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a83c5374-134e-415a-b45f-a15d8561bcbe:/dev/nvme0n1p3]
Sep 17 18:29:17.766739 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.766661    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:17.774093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.773978    2559 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:17.772744842 +0000 UTC m=+0.322968721 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097007 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29bda822252ded961bf61083529295 SystemUUID:ec29bda8-2225-2ded-961b-f61083529295 BootID:9dd97197-e2d6-4cfb-835e-9cd1e2ce66ed Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:85:68:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:85:68:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:54:a6:ad:d0:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:17.774093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.774082    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:17.774220 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.774200    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:17.777484 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.777460    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:17.777618 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.777488    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:17.778506 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.778495    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:17.778544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.778508    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:17.778544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.778516    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:17.779048 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.779032    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:17.779247 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.779235    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:17.779322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.779314    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:17.781919 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.781908    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:17.782031 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782022    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:17.782064 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782049    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:17.782095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782059    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:17.782095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782078    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:17.782095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782087    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:17.782095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782096    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:17.782448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782436    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:17.782482 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.782455    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:17.784618 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.784599    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:17.786724 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.786710    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:17.786776 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.786740    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:17.788557 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788545    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788563    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788570    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788575    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788584    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788590    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788603    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:17.788608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788608    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:17.788869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788616    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:17.788869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788622    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:17.788869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788640    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:17.788869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.788646    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:17.789324 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.789313    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:17.789324 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.789322    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:17.792919 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.792905    2559 server.go:1268] "Started kubelet"
Sep 17 18:29:17.793009 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.792982    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:17.793168 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.793022    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:17.793227 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.793194    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:17.793887 ip-10-0-143-176 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:17.794197 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.794174    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:17.795146 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.795060    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:17.795146 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.795091    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:17.795269 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.795178    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:17.796846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.796799    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:17.800093 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.800074    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:17.801763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.801747    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:17.801952 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.801046    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-176.ec2.internal.18d62f29a28108c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-176.ec2.internal,UID:ip-10-0-143-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-176.ec2.internal,},FirstTimestamp:2026-09-17 18:29:17.792880833 +0000 UTC m=+0.343104711,LastTimestamp:2026-09-17 18:29:17.792880833 +0000 UTC m=+0.343104711,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-176.ec2.internal,}"
Sep 17 18:29:17.802328 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.802311    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:17.802540 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.802520    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:17.802890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.802872    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:17.802890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.802873    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:17.803005 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.802904    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:17.803005 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.802995    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:17.803112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.803016    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:17.803539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.803521    2559 factory.go:55] Registering systemd factory
Sep 17 18:29:17.803595 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.803546    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:17.803787 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.803767    2559 factory.go:153] Registering CRI-O factory
Sep 17 18:29:17.803787 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.803781    2559 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:17.804090 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.804071    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:17.804146 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.804102    2559 factory.go:103] Registering Raw factory
Sep 17 18:29:17.804146 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.804121    2559 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:17.804545 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.804529    2559 manager.go:349] Starting recovery of all containers
Sep 17 18:29:17.811820 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.811792    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:17.811949 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.811925    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:17.812490 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.812445    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:17.815943 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.815925    2559 manager.go:354] Recovery completed
Sep 17 18:29:17.820163 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.820072    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:17.825411 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.825386    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:17.825483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.825425    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:17.825483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.825436    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:17.826065 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826049    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:17.826065 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826060    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:17.826190 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826065    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:17.826190 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826094    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:17.826894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826877    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:17.826962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826910    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:17.826962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.826928    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:17.827332 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.827315    2559 policy_none.go:49] "None policy: Start"
Sep 17 18:29:17.827402 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.827336    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:17.827402 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.827350    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:17.827525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.827444    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-176.ec2.internal.18d62f29a47166dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-176.ec2.internal,UID:ip-10-0-143-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-176.ec2.internal,},FirstTimestamp:2026-09-17 18:29:17.82541078 +0000 UTC m=+0.375634658,LastTimestamp:2026-09-17 18:29:17.82541078 +0000 UTC m=+0.375634658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-176.ec2.internal,}"
Sep 17 18:29:17.828264 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.828254    2559 policy_none.go:47] "Start"
Sep 17 18:29:17.829542 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.829515    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-529bd"
Sep 17 18:29:17.836356 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.836284    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-176.ec2.internal.18d62f29a471b2b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-176.ec2.internal,UID:ip-10-0-143-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-176.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-176.ec2.internal,},FirstTimestamp:2026-09-17 18:29:17.825430193 +0000 UTC m=+0.375654071,LastTimestamp:2026-09-17 18:29:17.825430193 +0000 UTC m=+0.375654071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-176.ec2.internal,}"
Sep 17 18:29:17.838312 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.838297    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-529bd"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860241    2559 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.860278    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860288    2559 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860531    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860546    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860614    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860685    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.860693    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.861496    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:17.878085 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.861539    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:17.934780 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.934761    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:17.934905 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.934817    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:17.934905 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.934858    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:17.935002 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.934961    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:29:17.937276 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.937256    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:17.961492 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.961463    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:17.964766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.964149    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:17.964766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.964207    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:17.964766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.964223    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:17.964766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.964252    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-176.ec2.internal"
Sep 17 18:29:17.973804 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:17.973759    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-176.ec2.internal"
Sep 17 18:29:17.973804 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:17.973782    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-176.ec2.internal\": node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.006160 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.006139    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.035659 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.035630    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"]
Sep 17 18:29:18.035785 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.035706    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:18.037436 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.037420    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:18.037539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.037453    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:18.037539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.037477    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:18.039927 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.039912    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:18.040037 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.040023    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.040082 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.040058    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:18.043340 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043323    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:18.043414 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043348    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:18.043414 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043357    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:18.043414 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043368    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:18.043414 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043390    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:18.043414 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.043403    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:18.045638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.045620    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.045726 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.045646    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:18.047461 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.047446    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:18.047526 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.047472    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:18.047526 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.047482    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:18.072507 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.072490    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-176.ec2.internal\" not found" node="ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.078697 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.078678    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-176.ec2.internal\" not found" node="ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.104267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.104236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53df0d23cf6bfbf0621c183fbd5dbe70-config\") pod \"kube-apiserver-proxy-ip-10-0-143-176.ec2.internal\" (UID: \"53df0d23cf6bfbf0621c183fbd5dbe70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.104368 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.104272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.104368 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.104297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.106490 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.106476    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.204538 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53df0d23cf6bfbf0621c183fbd5dbe70-config\") pod \"kube-apiserver-proxy-ip-10-0-143-176.ec2.internal\" (UID: \"53df0d23cf6bfbf0621c183fbd5dbe70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.204538 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.204700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.204700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.204700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53df0d23cf6bfbf0621c183fbd5dbe70-config\") pod \"kube-apiserver-proxy-ip-10-0-143-176.ec2.internal\" (UID: \"53df0d23cf6bfbf0621c183fbd5dbe70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.204700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.204673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a7acdf147264272f5b5fc518129f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal\" (UID: \"62a7acdf147264272f5b5fc518129f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.206569 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.206550    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.307271 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.307245    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.376470 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.376445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.382714 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.381949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:18.408448 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.408279    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.508697 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.508649    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.609611 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.609535    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.710057 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.710026    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.731476 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.731446    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:18.745260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.745236    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:18.745371 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.745353    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:18.745427 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.745399    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:18.810486 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.810461    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.843176 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.843109    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:18 +0000 UTC" deadline="2028-03-03 00:25:52.524787767 +0000 UTC"
Sep 17 18:29:18.843176 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.843172    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12773h56m33.681621128s"
Sep 17 18:29:18.910855 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:18.910814    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:18.948770 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.948743    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:18.949834 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.949816    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:18.949962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.949870    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:18.949962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:18.949887    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:19.011096 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.011070    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-176.ec2.internal\" not found"
Sep 17 18:29:19.013917 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.013893    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:19.043770 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:19.043742    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53df0d23cf6bfbf0621c183fbd5dbe70.slice/crio-17b5a54b802105f50772b04d52f36c06340141d10d87653f4ff3155e85cfc228 WatchSource:0}: Error finding container 17b5a54b802105f50772b04d52f36c06340141d10d87653f4ff3155e85cfc228: Status 404 returned error can't find the container with id 17b5a54b802105f50772b04d52f36c06340141d10d87653f4ff3155e85cfc228
Sep 17 18:29:19.044021 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:19.044002    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a7acdf147264272f5b5fc518129f30.slice/crio-15a6e687572800d36263e760f5853fa0bbf08f682a4be58c52e93f06ad100d3d WatchSource:0}: Error finding container 15a6e687572800d36263e760f5853fa0bbf08f682a4be58c52e93f06ad100d3d: Status 404 returned error can't find the container with id 15a6e687572800d36263e760f5853fa0bbf08f682a4be58c52e93f06ad100d3d
Sep 17 18:29:19.045741 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.045718    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:19.050890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.050871    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:19.104273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.104244    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:19.117808 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.117790    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:19.118806 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.118794    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal"
Sep 17 18:29:19.128017 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.128000    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:19.783975 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.783936    2559 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:19.791814 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.791779    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:19.793356 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.793319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-42qln","kube-system/global-pull-secret-syncer-rf6t6","kube-system/konnectivity-agent-lmflg","openshift-dns/node-resolver-4lw8h","openshift-multus/multus-additional-cni-plugins-fbrlc","openshift-network-operator/iptables-alerter-5nd2b","openshift-ovn-kubernetes/ovnkube-node-v52kc","kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9","openshift-cluster-node-tuning-operator/tuned-rz9l8","openshift-image-registry/node-ca-rl9dv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal","openshift-multus/multus-6mxw8","openshift-multus/network-metrics-daemon-mtf79"]
Sep 17 18:29:19.796039 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.796016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.799410 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.798825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.799410 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.798938    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:19.799410 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.799260    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.799410 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.799272    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sc5jp\""
Sep 17 18:29:19.799410 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.799372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.802831 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.802287    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:19.802831 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.802430    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.804887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.805898    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.805942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.806354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.806564    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:19.806932 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.806792    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crgc6\""
Sep 17 18:29:19.807763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.807039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.807763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.807051    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:19.807763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.807359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnch9\""
Sep 17 18:29:19.807945 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.807928    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:19.808292 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.808130    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:19.808292 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.808213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.811101 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.811080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:19.812184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812165    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.812360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812209    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:19.812360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:19.812360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812240    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.812360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812343    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:19.812758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.812737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9r68\""
Sep 17 18:29:19.813368 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.813350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.813657 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.813636    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.813657 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.813651    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mpfb8\""
Sep 17 18:29:19.814005 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.813978    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:19.814702 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.814685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:19.815328 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.814983    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:19.815429 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815346    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:19.815836 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-var-lib-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.815964 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.815964 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.815964 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-config\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.815978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-script-lib\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34eb689d-dd14-41a6-a163-84c3650a3d1b-konnectivity-ca\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lkk\" (UniqueName: \"kubernetes.io/projected/3203cff7-0339-47b7-9e6a-75e92c7530dc-kube-api-access-z5lkk\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-systemd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcv4\" (UniqueName: \"kubernetes.io/projected/8fe2d461-8d80-4cb2-86fd-436d591bdafc-kube-api-access-sxcv4\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.816133 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-systemd-units\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-bin\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-netns\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-netd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-slash\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-etc-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-kubelet\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-node-log\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.816448 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-log-socket\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34eb689d-dd14-41a6-a163-84c3650a3d1b-agent-certs\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prz2\" (UniqueName: \"kubernetes.io/projected/38b3f519-272f-4c6a-8cd1-f78ed4884548-kube-api-access-8prz2\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe2d461-8d80-4cb2-86fd-436d591bdafc-hosts-file\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-ovn\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-env-overrides\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe2d461-8d80-4cb2-86fd-436d591bdafc-tmp-dir\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.817018 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.816675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovn-node-metrics-cert\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.817817 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.817792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.820492 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.820459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.820602 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.820587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-crrgt\""
Sep 17 18:29:19.821049 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.820892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:19.821297 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.821274    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.821379 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.821364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.824112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.824091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:19.825036 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.824972    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.825136 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.825055    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.825452 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.825299    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlgmz\""
Sep 17 18:29:19.826887 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.826866    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:19.827170 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.827151    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b8p2k\""
Sep 17 18:29:19.827393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.827364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.827479 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.827421    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:19.827534 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.827474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:19.829393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.829372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:19.829512 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.829442    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:19.829723 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.829707    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:19.830100 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.829869    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vgjbj\""
Sep 17 18:29:19.846408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.846385    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrsb8"
Sep 17 18:29:19.854477 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.854425    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrsb8"
Sep 17 18:29:19.904556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.904529    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:19.916931 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.916898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe2d461-8d80-4cb2-86fd-436d591bdafc-hosts-file\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7aab32f-502e-4264-83b1-715e9941aa3e-host-slash\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-k8s-cni-cncf-io\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923677    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe2d461-8d80-4cb2-86fd-436d591bdafc-hosts-file\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-multus\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-cnibin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-cni-binary-copy\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-bin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.923896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.923964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34eb689d-dd14-41a6-a163-84c3650a3d1b-konnectivity-ca\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-host\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lkk\" (UniqueName: \"kubernetes.io/projected/3203cff7-0339-47b7-9e6a-75e92c7530dc-kube-api-access-z5lkk\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-systemd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-systemd-units\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-cnibin\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-netns\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-netd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lhd\" (UniqueName: \"kubernetes.io/projected/b7aab32f-502e-4264-83b1-715e9941aa3e-kube-api-access-26lhd\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-etc-tuned\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-netns\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-registration-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-systemd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-run-netns\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-kubernetes\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-systemd-units\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-run\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34eb689d-dd14-41a6-a163-84c3650a3d1b-konnectivity-ca\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.924867    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:19.924926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-tmp\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.924960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-netd\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:19.924967    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.424942128 +0000 UTC m=+2.975166019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-node-log\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-log-socket\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34eb689d-dd14-41a6-a163-84c3650a3d1b-agent-certs\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-log-socket\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-node-log\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-modprobe-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-host\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-system-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prz2\" (UniqueName: \"kubernetes.io/projected/38b3f519-272f-4c6a-8cd1-f78ed4884548-kube-api-access-8prz2\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-conf\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-sys\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4l66\" (UniqueName: \"kubernetes.io/projected/373755b8-36da-43cd-8352-743f40e40afa-kube-api-access-m4l66\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925643    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:19.925824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925694    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-multus-daemon-config\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbhr7\" (UniqueName: \"kubernetes.io/projected/337523de-e49b-4985-9e89-8f641bdd018e-kube-api-access-rbhr7\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-ovn\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-env-overrides\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-serviceca\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.925940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-run-ovn\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-multus-certs\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe2d461-8d80-4cb2-86fd-436d591bdafc-tmp-dir\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovn-node-metrics-cert\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-kubelet-config\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-env-overrides\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2dx\" (UniqueName: \"kubernetes.io/projected/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-kube-api-access-vm2dx\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-lib-modules\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-var-lib-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-config\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-script-lib\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926666    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-systemd\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.926690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-conf-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe2d461-8d80-4cb2-86fd-436d591bdafc-tmp-dir\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-dbus\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcv4\" (UniqueName: \"kubernetes.io/projected/8fe2d461-8d80-4cb2-86fd-436d591bdafc-kube-api-access-sxcv4\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.926975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-var-lib-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-bin\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvpg\" (UniqueName: \"kubernetes.io/projected/179ac4a2-1135-45a2-9880-881b916e7b99-kube-api-access-ptvpg\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-socket-dir-parent\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-hostroot\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-socket-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-var-lib-kubelet\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-etc-kubernetes\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-config\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ktc\" (UniqueName: \"kubernetes.io/projected/cd2f3550-21bf-4f33-8a83-c2c84667900a-kube-api-access-g2ktc\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.927609 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-slash\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-slash\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-etc-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927570    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-etc-openvswitch\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-kubelet\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovnkube-script-lib\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-kubelet\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-cni-bin\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7aab32f-502e-4264-83b1-715e9941aa3e-iptables-alerter-script\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3203cff7-0339-47b7-9e6a-75e92c7530dc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-device-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.927993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-sys-fs\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.928044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-kubelet\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.928072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-system-cni-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.928201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:19.928522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.928237    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:19.929352 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.928269    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysconfig\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:19.930729 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.930608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3203cff7-0339-47b7-9e6a-75e92c7530dc-ovn-node-metrics-cert\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.930838 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.930730    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34eb689d-dd14-41a6-a163-84c3650a3d1b-agent-certs\") pod \"konnectivity-agent-lmflg\" (UID: \"34eb689d-dd14-41a6-a163-84c3650a3d1b\") " pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:19.934466 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.934437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lkk\" (UniqueName: \"kubernetes.io/projected/3203cff7-0339-47b7-9e6a-75e92c7530dc-kube-api-access-z5lkk\") pod \"ovnkube-node-v52kc\" (UID: \"3203cff7-0339-47b7-9e6a-75e92c7530dc\") " pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:19.936252 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.936231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcv4\" (UniqueName: \"kubernetes.io/projected/8fe2d461-8d80-4cb2-86fd-436d591bdafc-kube-api-access-sxcv4\") pod \"node-resolver-4lw8h\" (UID: \"8fe2d461-8d80-4cb2-86fd-436d591bdafc\") " pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:19.936641 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.936618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prz2\" (UniqueName: \"kubernetes.io/projected/38b3f519-272f-4c6a-8cd1-f78ed4884548-kube-api-access-8prz2\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:19.940102 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.940061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal" event={"ID":"62a7acdf147264272f5b5fc518129f30","Type":"ContainerStarted","Data":"15a6e687572800d36263e760f5853fa0bbf08f682a4be58c52e93f06ad100d3d"}
Sep 17 18:29:19.941487 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:19.941460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal" event={"ID":"53df0d23cf6bfbf0621c183fbd5dbe70","Type":"ContainerStarted","Data":"17b5a54b802105f50772b04d52f36c06340141d10d87653f4ff3155e85cfc228"}
Sep 17 18:29:20.029461 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7aab32f-502e-4264-83b1-715e9941aa3e-iptables-alerter-script\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-device-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-sys-fs\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-kubelet\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-system-cni-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-sys-fs\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-kubelet\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-device-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.029663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysconfig\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-system-cni-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029707    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysconfig\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7aab32f-502e-4264-83b1-715e9941aa3e-host-slash\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-k8s-cni-cncf-io\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-multus\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7aab32f-502e-4264-83b1-715e9941aa3e-host-slash\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-cnibin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-k8s-cni-cncf-io\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-cni-binary-copy\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-multus\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-bin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-cnibin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-var-lib-cni-bin\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-host\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.030208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.029988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-cnibin\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lhd\" (UniqueName: \"kubernetes.io/projected/b7aab32f-502e-4264-83b1-715e9941aa3e-kube-api-access-26lhd\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-etc-tuned\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-netns\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-registration-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-kubernetes\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-run\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-tmp\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7aab32f-502e-4264-83b1-715e9941aa3e-iptables-alerter-script\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-modprobe-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-host\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-modprobe-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-d\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-system-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-host\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-netns\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-conf\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-registration-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-sys\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-host\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4l66\" (UniqueName: \"kubernetes.io/projected/373755b8-36da-43cd-8352-743f40e40afa-kube-api-access-m4l66\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-sys\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-multus-daemon-config\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbhr7\" (UniqueName: \"kubernetes.io/projected/337523de-e49b-4985-9e89-8f641bdd018e-kube-api-access-rbhr7\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.030484    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-serviceca\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-cni-binary-copy\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.030530    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.530514542 +0000 UTC m=+3.080738408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-multus-certs\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.031982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-kubelet-config\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-host-run-multus-certs\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030600    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2dx\" (UniqueName: \"kubernetes.io/projected/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-kube-api-access-vm2dx\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-lib-modules\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-systemd\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-kubelet-config\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-conf-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-dbus\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvpg\" (UniqueName: \"kubernetes.io/projected/179ac4a2-1135-45a2-9880-881b916e7b99-kube-api-access-ptvpg\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-socket-dir-parent\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-sysctl-conf\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-hostroot\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-hostroot\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-conf-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-cnibin\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.030969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-systemd\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.032849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-lib-modules\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337523de-e49b-4985-9e89-8f641bdd018e-multus-daemon-config\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-multus-socket-dir-parent\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-etc-kubernetes\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-system-cni-dir\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-run\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/923b92fd-9415-43ec-ab08-465f6cd44407-dbus\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2f3550-21bf-4f33-8a83-c2c84667900a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-socket-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-var-lib-kubelet\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-etc-kubernetes\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/179ac4a2-1135-45a2-9880-881b916e7b99-socket-dir\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ktc\" (UniqueName: \"kubernetes.io/projected/cd2f3550-21bf-4f33-8a83-c2c84667900a-kube-api-access-g2ktc\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-serviceca\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337523de-e49b-4985-9e89-8f641bdd018e-etc-kubernetes\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.033793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/373755b8-36da-43cd-8352-743f40e40afa-var-lib-kubelet\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.034746 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.031958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2f3550-21bf-4f33-8a83-c2c84667900a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.034746 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.034492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-tmp\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.036463 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.036440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/373755b8-36da-43cd-8352-743f40e40afa-etc-tuned\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.040370 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.040348    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:20.040473 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.040374    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:20.040473 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.040388    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.040473 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.040462    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.540442768 +0000 UTC m=+3.090666636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.041540 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.041516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ktc\" (UniqueName: \"kubernetes.io/projected/cd2f3550-21bf-4f33-8a83-c2c84667900a-kube-api-access-g2ktc\") pod \"multus-additional-cni-plugins-fbrlc\" (UID: \"cd2f3550-21bf-4f33-8a83-c2c84667900a\") " pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.041830 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.041801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2dx\" (UniqueName: \"kubernetes.io/projected/ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6-kube-api-access-vm2dx\") pod \"node-ca-rl9dv\" (UID: \"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6\") " pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.043051 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.043022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lhd\" (UniqueName: \"kubernetes.io/projected/b7aab32f-502e-4264-83b1-715e9941aa3e-kube-api-access-26lhd\") pod \"iptables-alerter-5nd2b\" (UID: \"b7aab32f-502e-4264-83b1-715e9941aa3e\") " pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.043945 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.043920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbhr7\" (UniqueName: \"kubernetes.io/projected/337523de-e49b-4985-9e89-8f641bdd018e-kube-api-access-rbhr7\") pod \"multus-6mxw8\" (UID: \"337523de-e49b-4985-9e89-8f641bdd018e\") " pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.044355 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.044330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvpg\" (UniqueName: \"kubernetes.io/projected/179ac4a2-1135-45a2-9880-881b916e7b99-kube-api-access-ptvpg\") pod \"aws-ebs-csi-driver-node-pm6p9\" (UID: \"179ac4a2-1135-45a2-9880-881b916e7b99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.044826 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.044805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4l66\" (UniqueName: \"kubernetes.io/projected/373755b8-36da-43cd-8352-743f40e40afa-kube-api-access-m4l66\") pod \"tuned-rz9l8\" (UID: \"373755b8-36da-43cd-8352-743f40e40afa\") " pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.111297 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.111258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lw8h"
Sep 17 18:29:20.120063 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.120026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:20.130042 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.130008    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:20.134883 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.134774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fbrlc"
Sep 17 18:29:20.145885 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.145863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5nd2b"
Sep 17 18:29:20.153877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.153854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9"
Sep 17 18:29:20.161846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.161809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rz9l8"
Sep 17 18:29:20.169824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.169804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rl9dv"
Sep 17 18:29:20.173385 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.173363    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:20.175865 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.175845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mxw8"
Sep 17 18:29:20.434900 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.434796    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:20.435082 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.434959    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:20.435082 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.435045    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:21.435021893 +0000 UTC m=+3.985245763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:20.535562 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.535521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:20.535771 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.535712    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.535823 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.535800    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:21.535780305 +0000 UTC m=+4.086004194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.635977 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.635940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:20.636148 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.636108    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:20.636148 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.636132    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:20.636148 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.636146    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.636303 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.636200    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:21.63618653 +0000 UTC m=+4.186410397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.857611 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.857572    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:20 +0000 UTC" deadline="2028-06-02 16:24:09.238920332 +0000 UTC"
Sep 17 18:29:20.857611 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.857608    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14973h54m48.381316989s"
Sep 17 18:29:20.935232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:20.935195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:20.935393 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:20.935335    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:20.959267 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.959162    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337523de_e49b_4985_9e89_8f641bdd018e.slice/crio-837eb84e06557e7d81cae182850aa0803706a744e82c7a2514a55e5032ecb87f WatchSource:0}: Error finding container 837eb84e06557e7d81cae182850aa0803706a744e82c7a2514a55e5032ecb87f: Status 404 returned error can't find the container with id 837eb84e06557e7d81cae182850aa0803706a744e82c7a2514a55e5032ecb87f
Sep 17 18:29:20.961009 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.960980    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34eb689d_dd14_41a6_a163_84c3650a3d1b.slice/crio-226a5866f9801a08e2e1353a9a932a29a1cc396b663893b723ff443b9f498d15 WatchSource:0}: Error finding container 226a5866f9801a08e2e1353a9a932a29a1cc396b663893b723ff443b9f498d15: Status 404 returned error can't find the container with id 226a5866f9801a08e2e1353a9a932a29a1cc396b663893b723ff443b9f498d15
Sep 17 18:29:20.961679 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.961630    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe2d461_8d80_4cb2_86fd_436d591bdafc.slice/crio-8d46bf94928c4bb24d6bd99f830e6b9fbd89e62c2931d116123b7a31c43eb8d6 WatchSource:0}: Error finding container 8d46bf94928c4bb24d6bd99f830e6b9fbd89e62c2931d116123b7a31c43eb8d6: Status 404 returned error can't find the container with id 8d46bf94928c4bb24d6bd99f830e6b9fbd89e62c2931d116123b7a31c43eb8d6
Sep 17 18:29:20.962826 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.962794    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179ac4a2_1135_45a2_9880_881b916e7b99.slice/crio-b531cb95a0aea4781a77fc7cb1332ce45b4b448212ace054383ed563e7de029f WatchSource:0}: Error finding container b531cb95a0aea4781a77fc7cb1332ce45b4b448212ace054383ed563e7de029f: Status 404 returned error can't find the container with id b531cb95a0aea4781a77fc7cb1332ce45b4b448212ace054383ed563e7de029f
Sep 17 18:29:20.964470 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.964452    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3203cff7_0339_47b7_9e6a_75e92c7530dc.slice/crio-83b17cabf0c7f97d2cdbb5e1ed588fd7ecd2d7af7ea35c6a56eb8187c54e4c82 WatchSource:0}: Error finding container 83b17cabf0c7f97d2cdbb5e1ed588fd7ecd2d7af7ea35c6a56eb8187c54e4c82: Status 404 returned error can't find the container with id 83b17cabf0c7f97d2cdbb5e1ed588fd7ecd2d7af7ea35c6a56eb8187c54e4c82
Sep 17 18:29:20.968315 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.966036    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373755b8_36da_43cd_8352_743f40e40afa.slice/crio-fb32a50035f2a542acd94eb19b5bb3fac5257a8a868be29d649c6db2e1f8ffc8 WatchSource:0}: Error finding container fb32a50035f2a542acd94eb19b5bb3fac5257a8a868be29d649c6db2e1f8ffc8: Status 404 returned error can't find the container with id fb32a50035f2a542acd94eb19b5bb3fac5257a8a868be29d649c6db2e1f8ffc8
Sep 17 18:29:20.972690 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:20.972589    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1cbb8d_74a1_4b94_82c2_90d6dbec2ed6.slice/crio-f7281433d6181e12968de9a2543b286f814ede90f09d6610e9d22dd6cb1e468e WatchSource:0}: Error finding container f7281433d6181e12968de9a2543b286f814ede90f09d6610e9d22dd6cb1e468e: Status 404 returned error can't find the container with id f7281433d6181e12968de9a2543b286f814ede90f09d6610e9d22dd6cb1e468e
Sep 17 18:29:21.444187 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.443579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:21.444187 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.444032    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:21.444187 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.444106    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.444085601 +0000 UTC m=+5.994309471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:21.544559 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.544398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:21.544739 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.544593    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:21.544739 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.544657    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.544639442 +0000 UTC m=+6.094863327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:21.645722 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.645676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:21.645936 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.645901    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:21.645936 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.645923    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:21.645936 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.645937    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:21.646118 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.646010    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.645990788 +0000 UTC m=+6.196214664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:21.857891 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.857789    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:20 +0000 UTC" deadline="2028-03-31 01:58:11.176160582 +0000 UTC"
Sep 17 18:29:21.857891 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.857870    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13447h28m49.318295714s"
Sep 17 18:29:21.917341 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.917306    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:21.937066 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.937038    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:21.937271 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.937165    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:21.937840 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.937696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:21.937840 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:21.937802    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:21.951097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.950096    2559 generic.go:356] "Generic (PLEG): container finished" podID="62a7acdf147264272f5b5fc518129f30" containerID="8afffdca6580de34c6e74f7958a7f42ed91be94917c2de6057c14af9cd34b8a5" exitCode=0
Sep 17 18:29:21.951097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.950184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal" event={"ID":"62a7acdf147264272f5b5fc518129f30","Type":"ContainerDied","Data":"8afffdca6580de34c6e74f7958a7f42ed91be94917c2de6057c14af9cd34b8a5"}
Sep 17 18:29:21.976148 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.976111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerStarted","Data":"b06937a5b761e0756bf656285d92d2d77cbfe179381559bbb9db13f91eb7fd71"}
Sep 17 18:29:21.980515 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.980484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lw8h" event={"ID":"8fe2d461-8d80-4cb2-86fd-436d591bdafc","Type":"ContainerStarted","Data":"8d46bf94928c4bb24d6bd99f830e6b9fbd89e62c2931d116123b7a31c43eb8d6"}
Sep 17 18:29:21.987620 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:21.987597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal" event={"ID":"53df0d23cf6bfbf0621c183fbd5dbe70","Type":"ContainerStarted","Data":"f631b123dc696661f3ba248c0af8cbc93401f7b854877f5259b87cd2ee2f7a59"}
Sep 17 18:29:22.000754 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.000465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5nd2b" event={"ID":"b7aab32f-502e-4264-83b1-715e9941aa3e","Type":"ContainerStarted","Data":"5b36c508b4b28953d58b96a65487df01c53b1edf210dffdd439281855372bf05"}
Sep 17 18:29:22.025017 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.024967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rl9dv" event={"ID":"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6","Type":"ContainerStarted","Data":"f7281433d6181e12968de9a2543b286f814ede90f09d6610e9d22dd6cb1e468e"}
Sep 17 18:29:22.037919 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.037867    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"83b17cabf0c7f97d2cdbb5e1ed588fd7ecd2d7af7ea35c6a56eb8187c54e4c82"}
Sep 17 18:29:22.046094 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.046057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rz9l8" event={"ID":"373755b8-36da-43cd-8352-743f40e40afa","Type":"ContainerStarted","Data":"fb32a50035f2a542acd94eb19b5bb3fac5257a8a868be29d649c6db2e1f8ffc8"}
Sep 17 18:29:22.055678 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.055643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9" event={"ID":"179ac4a2-1135-45a2-9880-881b916e7b99","Type":"ContainerStarted","Data":"b531cb95a0aea4781a77fc7cb1332ce45b4b448212ace054383ed563e7de029f"}
Sep 17 18:29:22.079063 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.079027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lmflg" event={"ID":"34eb689d-dd14-41a6-a163-84c3650a3d1b","Type":"ContainerStarted","Data":"226a5866f9801a08e2e1353a9a932a29a1cc396b663893b723ff443b9f498d15"}
Sep 17 18:29:22.090303 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.090268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mxw8" event={"ID":"337523de-e49b-4985-9e89-8f641bdd018e","Type":"ContainerStarted","Data":"837eb84e06557e7d81cae182850aa0803706a744e82c7a2514a55e5032ecb87f"}
Sep 17 18:29:22.935707 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:22.935176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:22.935707 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:22.935325    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:23.115934 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.115724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal" event={"ID":"62a7acdf147264272f5b5fc518129f30","Type":"ContainerStarted","Data":"c6c650ca268e4717070667fc0541bb90c21e7916f4a770072aeeabddf03c07ee"}
Sep 17 18:29:23.135320 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.135248    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-176.ec2.internal" podStartSLOduration=4.135225246 podStartE2EDuration="4.135225246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:22.005157027 +0000 UTC m=+4.555380920" watchObservedRunningTime="2026-09-17 18:29:23.135225246 +0000 UTC m=+5.685449136"
Sep 17 18:29:23.461296 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.461252    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:23.461655 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.461590    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.461754 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.461720    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.461698543 +0000 UTC m=+10.011922417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.562387 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.562344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:23.562564 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.562540    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.562645 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.562608    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.562589858 +0000 UTC m=+10.112813732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.662853 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.662805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:23.663035 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.663016    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:23.663114 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.663039    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:23.663114 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.663052    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.663220 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.663117    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.663096991 +0000 UTC m=+10.213320859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.935689 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.935652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:23.935889 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.935789    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:23.936357 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:23.936241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:23.936357 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:23.936339    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:24.935134 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:24.935093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:24.935326 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:24.935269    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:25.936162 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:25.935637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:25.936162 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:25.935777    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:25.936162 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:25.936115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:25.936770 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:25.936227    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:26.936134 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:26.936097    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:26.936337 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:26.936244    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:27.496681 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:27.496637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:27.496887 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.496795    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:27.496977 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.496903    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.496857793 +0000 UTC m=+18.047081680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:27.598094 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:27.597458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:27.598094 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.597638    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:27.598094 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.597715    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.597696542 +0000 UTC m=+18.147920428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:27.699265 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:27.698691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:27.699265 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.698856    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:27.699265 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.698891    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:27.699265 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.698902    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:27.699265 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.698961    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.698943506 +0000 UTC m=+18.249167390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:27.941268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:27.941229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:27.941268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:27.941260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:27.941851 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.941364    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:27.941851 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:27.941516    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:28.936026 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:28.935991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:28.936218 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:28.936142    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:29.936044 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:29.936005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:29.936516 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:29.936142    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:29.936579 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:29.936517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:29.936650 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:29.936633    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:30.935614 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:30.935579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:30.935811 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:30.935708    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:31.936073 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:31.936039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:31.936632 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:31.936088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:31.936632 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:31.936180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:31.936632 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:31.936565    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:32.935451 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:32.935410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:32.935616 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:32.935550    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:33.935945 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:33.935912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:33.936288 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:33.935920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:33.936288 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:33.936018    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:33.936288 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:33.936099    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:34.935056 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:34.934993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:34.935267 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:34.935117    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:35.558116 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:35.558078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:35.558580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.558250    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.558580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.558318    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:51.558297227 +0000 UTC m=+34.108521097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.658845 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:35.658808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:35.658997 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.658964    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.659060 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.659026    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:51.659012462 +0000 UTC m=+34.209236327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.759398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:35.759361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:35.759586 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.759547    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:35.759586 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.759579    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:35.759696 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.759593    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:35.759696 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.759649    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:51.759632789 +0000 UTC m=+34.309856658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:35.935697 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:35.935602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:35.935697 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:35.935650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:35.935928 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.935738    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:35.935928 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:35.935877    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:36.935976 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:36.935939    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:36.936430 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:36.936103    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:37.935950 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:37.935910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:37.936147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:37.936012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:37.936147 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:37.936127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:37.936147 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:37.936017    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:38.935424 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:38.935375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:38.935588 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:38.935531    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:39.936442 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:39.936144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:39.936940 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:39.936221    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:39.936940 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:39.936518    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:39.936940 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:39.936589    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:40.163308 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.163218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rl9dv" event={"ID":"ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6","Type":"ContainerStarted","Data":"d8134aae74b5ec7fc606a02cb997dca001ee512c60f36cb7bad1c0556fde51fc"}
Sep 17 18:29:40.166312 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"7fbb21e42d1d53163bb66733119e0398e3c208407a5d6dfec6eb83ba5ac71d1c"}
Sep 17 18:29:40.166424 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"0b0d8ec23d1b0806bf7653941dd954b8dafdb142a888c51ecfe3e3bafcea4f0e"}
Sep 17 18:29:40.166424 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"6cfe69041617e067b0889360f0328a126ac397862ca100362b1cca6887fb7d7a"}
Sep 17 18:29:40.166503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"3f22532a9a0f0dfbcc79215f33db8d7ab77c1632cbf423c68f66cb917def16e3"}
Sep 17 18:29:40.166503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"1faba768a418b2386d88102878217c08f3aa6165735217e854edff3bde7f3189"}
Sep 17 18:29:40.166573 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.166520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"84936f1c9b2ec8994ca744b64c175d9217189239651f9c43952b00859d3ee158"}
Sep 17 18:29:40.167622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.167600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rz9l8" event={"ID":"373755b8-36da-43cd-8352-743f40e40afa","Type":"ContainerStarted","Data":"8e203be1eecc5ddbaee124f1354856852bc3a9f059f5dc8d23128e3ca8eb6a88"}
Sep 17 18:29:40.169024 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.169003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9" event={"ID":"179ac4a2-1135-45a2-9880-881b916e7b99","Type":"ContainerStarted","Data":"582fdf30d555209488b4ee3ac3c9955fdfbb3e0882045c486ac1e2cd59f17891"}
Sep 17 18:29:40.170263 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.170243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lmflg" event={"ID":"34eb689d-dd14-41a6-a163-84c3650a3d1b","Type":"ContainerStarted","Data":"e646d2e12af0620f5883e219169359707adcaebeb86b1112ef38bec6353ade0a"}
Sep 17 18:29:40.171661 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.171632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mxw8" event={"ID":"337523de-e49b-4985-9e89-8f641bdd018e","Type":"ContainerStarted","Data":"11fc84860bec9b3a0be3ec0033218934ec883ebd45f0068fa2c13bdb214b3586"}
Sep 17 18:29:40.173042 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.173020    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="c72214a7f51b94773716595ea2c205d2da314d011e68ecc0b9626886859ed8c5" exitCode=0
Sep 17 18:29:40.173132 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.173099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"c72214a7f51b94773716595ea2c205d2da314d011e68ecc0b9626886859ed8c5"}
Sep 17 18:29:40.174605 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.174584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lw8h" event={"ID":"8fe2d461-8d80-4cb2-86fd-436d591bdafc","Type":"ContainerStarted","Data":"cbecaa78d6872c7b4769e39d69376f89d25b0f5e42658d6868653e054261344e"}
Sep 17 18:29:40.180037 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.179999    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rl9dv" podStartSLOduration=2.612187174 podStartE2EDuration="21.179986698s" totalImagesPullingTime="18.567799524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.179523602 +0000 UTC m=+22.729747488" watchObservedRunningTime="2026-09-17 18:29:40.179986698 +0000 UTC m=+22.730210584"
Sep 17 18:29:40.180145 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.180078    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-176.ec2.internal" podStartSLOduration=21.180069967 podStartE2EDuration="21.180069967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:23.135976782 +0000 UTC m=+5.686200675" watchObservedRunningTime="2026-09-17 18:29:40.180069967 +0000 UTC m=+22.730293857"
Sep 17 18:29:40.225239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.225184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lmflg" podStartSLOduration=9.809295835 podStartE2EDuration="21.225167681s" totalImagesPullingTime="11.415871846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.197439831 +0000 UTC m=+22.747663718" watchObservedRunningTime="2026-09-17 18:29:40.225167681 +0000 UTC m=+22.775391567"
Sep 17 18:29:40.241209 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.241147    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4lw8h" podStartSLOduration=3.6650063250000002 podStartE2EDuration="22.241125357s" totalImagesPullingTime="18.576119032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.24067637 +0000 UTC m=+22.790900257" watchObservedRunningTime="2026-09-17 18:29:40.241125357 +0000 UTC m=+22.791349246"
Sep 17 18:29:40.257954 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.257899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rz9l8" podStartSLOduration=2.682190164 podStartE2EDuration="21.257878165s" totalImagesPullingTime="18.575688001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.25780469 +0000 UTC m=+22.808028576" watchObservedRunningTime="2026-09-17 18:29:40.257878165 +0000 UTC m=+22.808102046"
Sep 17 18:29:40.377218 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.377185    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:40.377848 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.377818    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:40.393188 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.393145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6mxw8" podStartSLOduration=2.762068768 podStartE2EDuration="21.393129607s" totalImagesPullingTime="18.631060839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.27460117 +0000 UTC m=+22.824825263" watchObservedRunningTime="2026-09-17 18:29:40.393129607 +0000 UTC m=+22.943353498"
Sep 17 18:29:40.935058 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:40.935030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:40.935235 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:40.935152    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:41.118446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.118416    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:41.179679 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.179636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9" event={"ID":"179ac4a2-1135-45a2-9880-881b916e7b99","Type":"ContainerStarted","Data":"0a1a6b1d9a10356179de2ec642dc6a1cd03722c9656ab6d4674551d66e43f33b"}
Sep 17 18:29:41.181679 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.181424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5nd2b" event={"ID":"b7aab32f-502e-4264-83b1-715e9941aa3e","Type":"ContainerStarted","Data":"27fdf963a05635cceea6c427dd33529296ffd9f3e49f4cc8267e16dc51688783"}
Sep 17 18:29:41.182071 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.182048    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:41.182581 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.182559    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lmflg"
Sep 17 18:29:41.197619 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.197530    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5nd2b" podStartSLOduration=3.630269342 podStartE2EDuration="22.19751113s" totalImagesPullingTime="18.567241788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:41.197301604 +0000 UTC m=+23.747525494" watchObservedRunningTime="2026-09-17 18:29:41.19751113 +0000 UTC m=+23.747735018"
Sep 17 18:29:41.874420 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.874275    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:41.118439306Z","UUID":"cdc3519d-e663-4022-b944-0f1b39fb1005","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:41.876301 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.876276    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:41.876438 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.876316    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:41.935147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.935121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:41.935280 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:41.935155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:41.935280 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:41.935239    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:41.935368 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:41.935325    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:42.188472 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:42.188437    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"91f068378d55a6785e6af41157f0625223ab1ad7e6836fb256a654a006427db2"}
Sep 17 18:29:42.190966 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:42.190935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9" event={"ID":"179ac4a2-1135-45a2-9880-881b916e7b99","Type":"ContainerStarted","Data":"daa67d31cead89d6ed78d67ad7f284fb14091db4f5e2aecf7fb845b0e65cedf9"}
Sep 17 18:29:42.935884 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:42.935852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:42.936055 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:42.935971    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:43.935333 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:43.935296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:43.935842 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:43.935296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:43.935842 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:43.935424    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:43.935842 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:43.935532    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:44.936230 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:44.936052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:44.936612 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:44.936311    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:45.199543 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.199465    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="460d1cceb05e775e9798091840517433714136ddddf8d67f2698372a6197b395" exitCode=0
Sep 17 18:29:45.199769 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.199549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"460d1cceb05e775e9798091840517433714136ddddf8d67f2698372a6197b395"}
Sep 17 18:29:45.203471 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.203436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" event={"ID":"3203cff7-0339-47b7-9e6a-75e92c7530dc","Type":"ContainerStarted","Data":"e20c003bfa4ab04769e9ff5c8e3734818742710358923c5853f3a2a1f8972949"}
Sep 17 18:29:45.203717 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.203695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:45.218974 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.218955    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:45.222893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.222859    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm6p9" podStartSLOduration=5.655128937 podStartE2EDuration="26.222846082s" totalImagesPullingTime="20.567717145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.238818004 +0000 UTC m=+24.789041892" watchObservedRunningTime="2026-09-17 18:29:45.222846082 +0000 UTC m=+27.773069960"
Sep 17 18:29:45.249048 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.248998    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc" podStartSLOduration=7.6196081079999995 podStartE2EDuration="26.248984174s" totalImagesPullingTime="18.629376066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:45.248870481 +0000 UTC m=+27.799094371" watchObservedRunningTime="2026-09-17 18:29:45.248984174 +0000 UTC m=+27.799208061"
Sep 17 18:29:45.936086 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.936053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:45.936242 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:45.936155    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:45.936242 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:45.936209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:45.936581 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:45.936313    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:46.205406 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.205216    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 18:29:46.205548 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.205381    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:46.219861 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.219820    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:46.295815 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.295786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:29:46.442793 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.442763    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42qln"]
Sep 17 18:29:46.442967 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.442951    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:46.443099 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:46.443061    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:46.443372 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.443319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtf79"]
Sep 17 18:29:46.443480 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.443427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:46.443537 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:46.443515    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:46.445323 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.445298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rf6t6"]
Sep 17 18:29:46.445422 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:46.445386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:46.445499 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:46.445477    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:47.209177 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:47.209148    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="7bec9fd868446e8b3377dc24333a1601af5f73a9a3039b10a6b35aa323dd160d" exitCode=0
Sep 17 18:29:47.209629 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:47.209216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"7bec9fd868446e8b3377dc24333a1601af5f73a9a3039b10a6b35aa323dd160d"}
Sep 17 18:29:47.936402 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:47.936375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:47.936526 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:47.936467    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:47.936526 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:47.936481    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:47.936526 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:47.936506    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:47.936688 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:47.936582    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:47.936743 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:47.936691    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:48.214427 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:48.214332    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="5fd9944b89e7a8a2092dd29a7caea3a0ed13457b6b9b23703592e32ff9ad49f1" exitCode=0
Sep 17 18:29:48.214427 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:48.214413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"5fd9944b89e7a8a2092dd29a7caea3a0ed13457b6b9b23703592e32ff9ad49f1"}
Sep 17 18:29:49.936061 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:49.936024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:49.936779 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:49.936153    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:49.936779 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:49.936019    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:49.936779 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:49.936270    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:49.936779 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:49.936332    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:49.936779 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:49.936415    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:51.574217 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.574026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:51.574612 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.574250    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:51.574612 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.574333    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs podName:38b3f519-272f-4c6a-8cd1-f78ed4884548 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:23.574311909 +0000 UTC m=+66.124535775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs") pod "network-metrics-daemon-mtf79" (UID: "38b3f519-272f-4c6a-8cd1-f78ed4884548") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:51.675093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.675055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:51.675268 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.675223    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:51.675331 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.675300    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret podName:923b92fd-9415-43ec-ab08-465f6cd44407 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:23.675282976 +0000 UTC m=+66.225506849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret") pod "global-pull-secret-syncer-rf6t6" (UID: "923b92fd-9415-43ec-ab08-465f6cd44407") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:51.776128 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.776089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:51.776308 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.776277    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:51.776308 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.776303    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:51.776395 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.776313    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5gqh for pod openshift-network-diagnostics/network-check-target-42qln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:51.776395 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.776365    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh podName:695d5ef7-dc4e-41b4-9aab-a9500c6dfc97 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:23.776351052 +0000 UTC m=+66.326574916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5gqh" (UniqueName: "kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh") pod "network-check-target-42qln" (UID: "695d5ef7-dc4e-41b4-9aab-a9500c6dfc97") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:51.935647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.935568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:51.935799 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.935570    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:51.935799 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.935721    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtf79" podUID="38b3f519-272f-4c6a-8cd1-f78ed4884548"
Sep 17 18:29:51.935922 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.935793    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rf6t6" podUID="923b92fd-9415-43ec-ab08-465f6cd44407"
Sep 17 18:29:51.935922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:51.935577    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:51.935922 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:51.935906    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42qln" podUID="695d5ef7-dc4e-41b4-9aab-a9500c6dfc97"
Sep 17 18:29:52.215593 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.215521    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-176.ec2.internal" event="NodeReady"
Sep 17 18:29:52.215722 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.215663    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:29:52.249957 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.249928    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xcgc8"]
Sep 17 18:29:52.260958 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.260926    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:29:52.261096 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.261044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.263521 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.263499    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 18:29:52.263788 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.263773    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.263854 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.263811    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kzxkn\""
Sep 17 18:29:52.263909 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.263851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 18:29:52.264247 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.264233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.268588 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.268305    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 18:29:52.281233 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.281211    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"]
Sep 17 18:29:52.281367 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.281343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.284145 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.284123    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:29:52.284480 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.284355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:29:52.284480 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.284402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-26p5d\""
Sep 17 18:29:52.284480 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.284407    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:29:52.291356 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.291340    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:29:52.307039 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.307019    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"]
Sep 17 18:29:52.307162 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.307138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.309701 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.309681    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:29:52.309816 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.309725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m49wr\""
Sep 17 18:29:52.309816 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.309772    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:29:52.323205 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.323182    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"]
Sep 17 18:29:52.323375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.323356    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.325722 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.325703    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 18:29:52.325838 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.325728    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gnw9j\""
Sep 17 18:29:52.325891 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.325874    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.326005 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.325989    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.326035 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.326001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 18:29:52.337420 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.337400    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"]
Sep 17 18:29:52.337504 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.337432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xcgc8"]
Sep 17 18:29:52.337504 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.337446    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:29:52.337504 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.337462    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d4456f768-xpfh4"]
Sep 17 18:29:52.337607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.337575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"
Sep 17 18:29:52.339822 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.339805    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.339822 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.339812    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rbhzb\""
Sep 17 18:29:52.339976 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.339863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.357901 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.357879    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"]
Sep 17 18:29:52.358017 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.357945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"]
Sep 17 18:29:52.358017 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.357971    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hskxl"]
Sep 17 18:29:52.358089 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.358047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.360700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360678    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 18:29:52.360813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360790    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.360891 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360790    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 18:29:52.360891 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 18:29:52.360975 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.361016 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.360969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 18:29:52.361238 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.361224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wk4f4\""
Sep 17 18:29:52.378056 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.378032    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"]
Sep 17 18:29:52.378190 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.378150    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.379660 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379743 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-serving-cert\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.379743 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379743 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-trusted-ca\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.379893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.379893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw25v\" (UniqueName: \"kubernetes.io/projected/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-kube-api-access-lw25v\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.380057 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdbl\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.380057 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-config\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.380057 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.379972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.380759 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.380742    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:29:52.380759 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.380756    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpkjn\""
Sep 17 18:29:52.380930 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.380743    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:29:52.399756 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.399638    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"]
Sep 17 18:29:52.399887 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.399849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.402441 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.402416    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 18:29:52.402554 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.402530    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-46fqs\""
Sep 17 18:29:52.402646 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.402582    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 18:29:52.402646 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.402626    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.402646 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.402626    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.423489 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.423461    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gmcpd"]
Sep 17 18:29:52.423617 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.423568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.425784 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.425768    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.426013 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.425994    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 18:29:52.426013 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.426011    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.426166 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.426050    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n9qvt\""
Sep 17 18:29:52.437879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.437861    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"]
Sep 17 18:29:52.437953 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.437901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.440865 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.440845    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.441009 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.440992    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 18:29:52.441061 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.440997    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.441098 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.441088    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 18:29:52.441348 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.441327    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pdp7s\""
Sep 17 18:29:52.447525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.447499    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 18:29:52.448813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.448767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:52.457875 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.457855    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"]
Sep 17 18:29:52.457989 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.457973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.460780 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.460755    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.460879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.460790    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.461039 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.461026    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 18:29:52.461095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.461079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dm7mx\""
Sep 17 18:29:52.461773 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.461758    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 18:29:52.478361 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.478307    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xs4w8"]
Sep 17 18:29:52.478449 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.478426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"
Sep 17 18:29:52.480306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.480416 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.480416 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw25v\" (UniqueName: \"kubernetes.io/projected/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-kube-api-access-lw25v\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.480514 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtl6w\" (UniqueName: \"kubernetes.io/projected/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-kube-api-access-rtl6w\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.480514 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknnz\" (UniqueName: \"kubernetes.io/projected/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-kube-api-access-hknnz\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.480607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.480607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclgf\" (UniqueName: \"kubernetes.io/projected/4fee1728-d0c7-452a-88da-1f97a178a7d1-kube-api-access-lclgf\") pod \"volume-data-source-validator-5c4db8bdf9-7qgvc\" (UID: \"4fee1728-d0c7-452a-88da-1f97a178a7d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"
Sep 17 18:29:52.480607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-serving-cert\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvv87\" (UniqueName: \"kubernetes.io/projected/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-kube-api-access-gvv87\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.480987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-config-volume\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-stats-auth\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-tmp-dir\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgmn\" (UniqueName: \"kubernetes.io/projected/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-kube-api-access-9qgmn\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdbl\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.481179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-default-certificate\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.481549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-config\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.481549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.481549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.481962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-config\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-serving-cert\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-trusted-ca\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-config\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.482306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.482309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lcz5c\""
Sep 17 18:29:52.482892 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.482364    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:52.482892 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.482374    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:29:52.482892 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.482418    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.982400331 +0000 UTC m=+35.532624200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:29:52.483045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.483031    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.483401 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.483107    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.483401 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.483230    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.486498 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.486477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-serving-cert\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.486598 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.486505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.486598 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.486574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.488716 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.488698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-trusted-ca\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.499399 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499382    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d4456f768-xpfh4"]
Sep 17 18:29:52.499481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499408    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"]
Sep 17 18:29:52.499481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hskxl"]
Sep 17 18:29:52.499481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gmcpd"]
Sep 17 18:29:52.499481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499466    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"]
Sep 17 18:29:52.499481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"]
Sep 17 18:29:52.499619 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499488    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.499619 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499493    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs4w8"]
Sep 17 18:29:52.499678 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.499643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"]
Sep 17 18:29:52.502352 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.502335    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:29:52.502893 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.502858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jr68f\""
Sep 17 18:29:52.503813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.503794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdbl\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.505211 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.505193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw25v\" (UniqueName: \"kubernetes.io/projected/cc9bafcc-d25f-4e55-8f3d-5eb321bbde11-kube-api-access-lw25v\") pod \"console-operator-fd4bb46c-xcgc8\" (UID: \"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11\") " pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.508938 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.508919    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:29:52.524048 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.524024    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:29:52.526338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.526317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.556341 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.556319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:52.570554 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.570532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:29:52.583548 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-config\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583562    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-tmp\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.583731    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtl6w\" (UniqueName: \"kubernetes.io/projected/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-kube-api-access-rtl6w\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.583767    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.583785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.083768742 +0000 UTC m=+35.633992627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.583877 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.583854    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.083810679 +0000 UTC m=+35.634034558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknnz\" (UniqueName: \"kubernetes.io/projected/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-kube-api-access-hknnz\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/caef9342-c962-42d4-b237-ed36f6d1a603-serving-cert\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.583988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclgf\" (UniqueName: \"kubernetes.io/projected/4fee1728-d0c7-452a-88da-1f97a178a7d1-kube-api-access-lclgf\") pod \"volume-data-source-validator-5c4db8bdf9-7qgvc\" (UID: \"4fee1728-d0c7-452a-88da-1f97a178a7d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62sj\" (UniqueName: \"kubernetes.io/projected/5588c483-4167-4b8d-8adc-eecfc98fd800-kube-api-access-h62sj\") pod \"network-check-source-7cbc45fddc-bt5vp\" (UID: \"5588c483-4167-4b8d-8adc-eecfc98fd800\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75347877-8dc6-45b1-be39-82372a6459ec-cert\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pq97\" (UniqueName: \"kubernetes.io/projected/d0ee7e92-446b-4fd2-817f-363399f71b8c-kube-api-access-7pq97\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5nz\" (UniqueName: \"kubernetes.io/projected/2c566714-79e3-4017-874b-8563302c1baa-kube-api-access-bg5nz\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-config\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzzf\" (UniqueName: \"kubernetes.io/projected/75347877-8dc6-45b1-be39-82372a6459ec-kube-api-access-dhzzf\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-snapshots\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.584460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvv87\" (UniqueName: \"kubernetes.io/projected/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-kube-api-access-gvv87\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-config-volume\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-stats-auth\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-tmp-dir\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgmn\" (UniqueName: \"kubernetes.io/projected/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-kube-api-access-9qgmn\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-default-certificate\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5l8q\" (UniqueName: \"kubernetes.io/projected/caef9342-c962-42d4-b237-ed36f6d1a603-kube-api-access-k5l8q\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.584628    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.584674    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.08465829 +0000 UTC m=+35.634882174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-tmp-dir\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.584974    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.08495707 +0000 UTC m=+35.635180935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:29:52.585375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-config-volume\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.586201 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.584994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d0ee7e92-446b-4fd2-817f-363399f71b8c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.586201 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.585025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-serving-cert\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.587195 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.587174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-serving-cert\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.587393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.587370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-default-certificate\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.587492 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.587481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.587674 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.587652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-stats-auth\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.594774 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.594750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtl6w\" (UniqueName: \"kubernetes.io/projected/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-kube-api-access-rtl6w\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:52.596006 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.595979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknnz\" (UniqueName: \"kubernetes.io/projected/ce4b99ae-7092-43b0-be9a-74dddc6a6d98-kube-api-access-hknnz\") pod \"service-ca-operator-57cdc4b768-j5z8v\" (UID: \"ce4b99ae-7092-43b0-be9a-74dddc6a6d98\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.596104 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.596083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvv87\" (UniqueName: \"kubernetes.io/projected/c0d6e2ef-a0b6-4517-ae26-54d0803e3599-kube-api-access-gvv87\") pod \"kube-storage-version-migrator-operator-55889f7b47-gs7jx\" (UID: \"c0d6e2ef-a0b6-4517-ae26-54d0803e3599\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.596423 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.596370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclgf\" (UniqueName: \"kubernetes.io/projected/4fee1728-d0c7-452a-88da-1f97a178a7d1-kube-api-access-lclgf\") pod \"volume-data-source-validator-5c4db8bdf9-7qgvc\" (UID: \"4fee1728-d0c7-452a-88da-1f97a178a7d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"
Sep 17 18:29:52.596909 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.596889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgmn\" (UniqueName: \"kubernetes.io/projected/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-kube-api-access-9qgmn\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:52.631760 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.631725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"
Sep 17 18:29:52.645162 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.644992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"
Sep 17 18:29:52.686135 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.686260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.686260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/caef9342-c962-42d4-b237-ed36f6d1a603-serving-cert\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.686260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62sj\" (UniqueName: \"kubernetes.io/projected/5588c483-4167-4b8d-8adc-eecfc98fd800-kube-api-access-h62sj\") pod \"network-check-source-7cbc45fddc-bt5vp\" (UID: \"5588c483-4167-4b8d-8adc-eecfc98fd800\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"
Sep 17 18:29:52.686260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75347877-8dc6-45b1-be39-82372a6459ec-cert\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pq97\" (UniqueName: \"kubernetes.io/projected/d0ee7e92-446b-4fd2-817f-363399f71b8c-kube-api-access-7pq97\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5nz\" (UniqueName: \"kubernetes.io/projected/2c566714-79e3-4017-874b-8563302c1baa-kube-api-access-bg5nz\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzzf\" (UniqueName: \"kubernetes.io/projected/75347877-8dc6-45b1-be39-82372a6459ec-kube-api-access-dhzzf\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-snapshots\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.686478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.686863 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5l8q\" (UniqueName: \"kubernetes.io/projected/caef9342-c962-42d4-b237-ed36f6d1a603-kube-api-access-k5l8q\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.686863 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d0ee7e92-446b-4fd2-817f-363399f71b8c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.686863 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.686584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-tmp\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.687020 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.686972    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:52.687068 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.687046    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.187026398 +0000 UTC m=+35.737250276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:52.687151 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.687127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.687398 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.687379    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:29:52.687519 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.687434    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.187418742 +0000 UTC m=+35.737642629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:29:52.687519 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.687430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-snapshots\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.687663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.687633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caef9342-c962-42d4-b237-ed36f6d1a603-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.687715 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.687697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef9342-c962-42d4-b237-ed36f6d1a603-tmp\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.687964 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.687946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d0ee7e92-446b-4fd2-817f-363399f71b8c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.689460 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.689440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75347877-8dc6-45b1-be39-82372a6459ec-cert\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.689519 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.689478    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/caef9342-c962-42d4-b237-ed36f6d1a603-serving-cert\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.695902 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.695737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62sj\" (UniqueName: \"kubernetes.io/projected/5588c483-4167-4b8d-8adc-eecfc98fd800-kube-api-access-h62sj\") pod \"network-check-source-7cbc45fddc-bt5vp\" (UID: \"5588c483-4167-4b8d-8adc-eecfc98fd800\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"
Sep 17 18:29:52.696029 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.695782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pq97\" (UniqueName: \"kubernetes.io/projected/d0ee7e92-446b-4fd2-817f-363399f71b8c-kube-api-access-7pq97\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:52.696029 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.695872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5l8q\" (UniqueName: \"kubernetes.io/projected/caef9342-c962-42d4-b237-ed36f6d1a603-kube-api-access-k5l8q\") pod \"insights-operator-6f9c644dd9-gmcpd\" (UID: \"caef9342-c962-42d4-b237-ed36f6d1a603\") " pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.697679 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.697649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzzf\" (UniqueName: \"kubernetes.io/projected/75347877-8dc6-45b1-be39-82372a6459ec-kube-api-access-dhzzf\") pod \"ingress-canary-xs4w8\" (UID: \"75347877-8dc6-45b1-be39-82372a6459ec\") " pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.698036 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.698015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5nz\" (UniqueName: \"kubernetes.io/projected/2c566714-79e3-4017-874b-8563302c1baa-kube-api-access-bg5nz\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:52.708461 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.708444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"
Sep 17 18:29:52.748310 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.748286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd"
Sep 17 18:29:52.797141 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.797111    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"
Sep 17 18:29:52.813416 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.813390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs4w8"
Sep 17 18:29:52.868239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.868209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xcgc8"]
Sep 17 18:29:52.989525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:52.989496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:52.989682 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.989668    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:52.989737 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.989689    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:29:52.989792 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:52.989757    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:53.989737106 +0000 UTC m=+36.539960988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:29:53.068278 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.068182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:53.090758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.090729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:53.090905 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.090768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:53.090967 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.090928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:53.090967 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.090931    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:29:53.091068 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.090946    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.090931888 +0000 UTC m=+36.641155753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:29:53.091068 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.090988    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:29:53.091068 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.090997    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.090985392 +0000 UTC m=+36.641209261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:29:53.091068 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.091020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:53.091068 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.091039    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.091024106 +0000 UTC m=+36.641247995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:29:53.091256 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.091075    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:53.091256 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.091099    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.091092132 +0000 UTC m=+36.641315997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:53.152520 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.152491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs4w8"]
Sep 17 18:29:53.158059 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.158035    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v"]
Sep 17 18:29:53.159328 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.159303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:53.169431 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.169407    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gmcpd"]
Sep 17 18:29:53.186113 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.186092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc"]
Sep 17 18:29:53.189571 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.189547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx"]
Sep 17 18:29:53.190738 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.190654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp"]
Sep 17 18:29:53.192070 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.192033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:53.192167 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.192094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:53.192239 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.192184    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:53.192239 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.192192    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:29:53.192326 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.192256    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.192236008 +0000 UTC m=+36.742459892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:53.192326 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:53.192281    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.192265196 +0000 UTC m=+36.742489081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:29:53.192606 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:53.192583    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5588c483_4167_4b8d_8adc_eecfc98fd800.slice/crio-4d316a5d0f945284900f132ed95e0a4db72469fab0372f1ad42964d992e70317 WatchSource:0}: Error finding container 4d316a5d0f945284900f132ed95e0a4db72469fab0372f1ad42964d992e70317: Status 404 returned error can't find the container with id 4d316a5d0f945284900f132ed95e0a4db72469fab0372f1ad42964d992e70317
Sep 17 18:29:53.193327 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:29:53.193303    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fee1728_d0c7_452a_88da_1f97a178a7d1.slice/crio-2cd70bffebb9084dbbbcf29ed126c8104ac74f9b8f21d608a8a70d0b031a7175 WatchSource:0}: Error finding container 2cd70bffebb9084dbbbcf29ed126c8104ac74f9b8f21d608a8a70d0b031a7175: Status 404 returned error can't find the container with id 2cd70bffebb9084dbbbcf29ed126c8104ac74f9b8f21d608a8a70d0b031a7175
Sep 17 18:29:53.226703 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.226668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" event={"ID":"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11","Type":"ContainerStarted","Data":"94e8f73d76d1397264ab6ec121e5dee4dfa02c3bf14e560329abda21aacd1c86"}
Sep 17 18:29:53.227714 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.227687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs4w8" event={"ID":"75347877-8dc6-45b1-be39-82372a6459ec","Type":"ContainerStarted","Data":"7cb932cef91b7f277ec5b0714bcc430c3f2d7ace7f891b693376d476fe686730"}
Sep 17 18:29:53.228772 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.228747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp" event={"ID":"5588c483-4167-4b8d-8adc-eecfc98fd800","Type":"ContainerStarted","Data":"4d316a5d0f945284900f132ed95e0a4db72469fab0372f1ad42964d992e70317"}
Sep 17 18:29:53.229747 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.229725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd" event={"ID":"caef9342-c962-42d4-b237-ed36f6d1a603","Type":"ContainerStarted","Data":"78b25f6e0754236194ab139470d96132f5d5fe2f60f243fbd16a675f1775b19e"}
Sep 17 18:29:53.230880 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.230854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v" event={"ID":"ce4b99ae-7092-43b0-be9a-74dddc6a6d98","Type":"ContainerStarted","Data":"bed783afab9f2ac142c4c9998b5486c72a4fbcb0ad3299d32d4e312d933d1f27"}
Sep 17 18:29:53.231986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.231967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc" event={"ID":"4fee1728-d0c7-452a-88da-1f97a178a7d1","Type":"ContainerStarted","Data":"2cd70bffebb9084dbbbcf29ed126c8104ac74f9b8f21d608a8a70d0b031a7175"}
Sep 17 18:29:53.232949 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.232932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx" event={"ID":"c0d6e2ef-a0b6-4517-ae26-54d0803e3599","Type":"ContainerStarted","Data":"1149c851b601d1075f42f7a3b42f96e10a25c4fe8c90d5a35932173e4302ff4c"}
Sep 17 18:29:53.235537 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.235517    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="d9eb12e6ff7d6d2d66e5a1ab120a92ae34531370e498447bd7ebff036d8d1b93" exitCode=0
Sep 17 18:29:53.235617 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.235543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"d9eb12e6ff7d6d2d66e5a1ab120a92ae34531370e498447bd7ebff036d8d1b93"}
Sep 17 18:29:53.243667 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.243649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:53.938804 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.938767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:29:53.939562 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.939257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:29:53.939683 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.939665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:29:53.941650 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.941629    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:29:53.943909 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.943888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:29:53.944340 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.944011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rcjbl\""
Sep 17 18:29:53.944340 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:53.944127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzhw6\""
Sep 17 18:29:54.001876 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.001845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:54.002150 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.002133    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:54.002246 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.002153    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:29:54.002246 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.002208    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.002190568 +0000 UTC m=+38.552414437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:29:54.050264 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.050182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.102931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.102979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.103162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.103202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.103413    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.103383527 +0000 UTC m=+38.653607400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.103845    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.103903    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.103884778 +0000 UTC m=+38.654108645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.103971    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.104002    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.103990911 +0000 UTC m=+38.654214778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.104392    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:29:54.104500 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.104457    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.104443261 +0000 UTC m=+38.654667187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:29:54.154251 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.154195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.204650 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.204397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:54.204650 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.204459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:54.204650 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.204566    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:54.204650 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.204642    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.204619021 +0000 UTC m=+38.754842891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:54.205185 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.204572    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:29:54.205185 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:54.205060    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.20504027 +0000 UTC m=+38.755264140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:29:54.248582 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.248458    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd2f3550-21bf-4f33-8a83-c2c84667900a" containerID="ac2f35b3237b4be5d212b3b6d23f9e5ceaa54861aeb1f4e4cc179607a8ec74ad" exitCode=0
Sep 17 18:29:54.248582 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.248527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerDied","Data":"ac2f35b3237b4be5d212b3b6d23f9e5ceaa54861aeb1f4e4cc179607a8ec74ad"}
Sep 17 18:29:54.398352 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.398293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.501622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:54.501526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.031303 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:55.031223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.396943 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:55.396824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.459366 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:55.459290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.025998 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.025966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:29:56.026142 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.026123    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:56.026142 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.026141    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:29:56.026239 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.026208    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.026186648 +0000 UTC m=+42.576410516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:29:56.126717 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.126674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:56.126717 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.126723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.126867    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.126877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.126917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.126941    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.126922388 +0000 UTC m=+42.677146268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.126940    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.127000    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.126984288 +0000 UTC m=+42.677208165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.127005    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.127022    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.127010433 +0000 UTC m=+42.677234300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:29:56.127525 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.127044    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.127030807 +0000 UTC m=+42.677254672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:29:56.227976 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.227938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:29:56.228155 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:56.228042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:29:56.228155 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.228102    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:56.228282 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.228181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.228159455 +0000 UTC m=+42.778383326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:29:56.228282 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.228185    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:29:56.228282 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:29:56.228262    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.22822359 +0000 UTC m=+42.778447472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:29:57.825337 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:57.825305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.925774 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:57.925736    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.382412 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:58.382372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.457311 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:58.457269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.195702 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:59.195667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.299858 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:29:59.299765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:00.060943 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.060911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:00.061086 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.061068    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:00.061122 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.061087    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:30:00.061170 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.061140    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.061124997 +0000 UTC m=+50.611348866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:30:00.161923 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.161892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:00.162024 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.161931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:00.162069 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.162029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:30:00.162069 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162038    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:00.162069 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.162054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:00.162205 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162094    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:00.162205 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162098    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.162084292 +0000 UTC m=+50.712308156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:30:00.162205 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.162144408 +0000 UTC m=+50.712368287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:00.162205 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162178    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.162168147 +0000 UTC m=+50.712392019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:00.162205 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162193    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:00.162392 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.162260    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.162240574 +0000 UTC m=+50.712464451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:00.263225 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.263197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:30:00.263580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.263357    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:00.263580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:00.263395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:30:00.263580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.263428    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.263405708 +0000 UTC m=+50.813629589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:30:00.263580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.263476    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:00.263580 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:00.263518    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.263508022 +0000 UTC m=+50.813731887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:01.272119 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.272078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v" event={"ID":"ce4b99ae-7092-43b0-be9a-74dddc6a6d98","Type":"ContainerStarted","Data":"5811de7cc3244ce424bb9ee9ae0ddc51cb384e2bfc77520683a3f578fdb9a7d1"}
Sep 17 18:30:01.273770 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.273742    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc" event={"ID":"4fee1728-d0c7-452a-88da-1f97a178a7d1","Type":"ContainerStarted","Data":"93ed4a02d678d41bdf37fc77b3fae5d04d2c1eb7ff8e7f90d660ef4f2286b518"}
Sep 17 18:30:01.275517 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.275489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx" event={"ID":"c0d6e2ef-a0b6-4517-ae26-54d0803e3599","Type":"ContainerStarted","Data":"e01f08d8cfb8296a2e544c1340e01f4a90e28d18e15661ea26432fe30a66ac3f"}
Sep 17 18:30:01.279598 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.279576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" event={"ID":"cd2f3550-21bf-4f33-8a83-c2c84667900a","Type":"ContainerStarted","Data":"f8bf3bf750e9451e5ff1ec6cca5be22a5a3b02a24f99ef86cb29f6ea0e81b02b"}
Sep 17 18:30:01.281330 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.281314    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/0.log"
Sep 17 18:30:01.281459 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.281346    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc9bafcc-d25f-4e55-8f3d-5eb321bbde11" containerID="a0657c1eabf1fadcba6249c5623a3b599f852140407791dd569eceb7f00ca66d" exitCode=255
Sep 17 18:30:01.281459 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.281417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" event={"ID":"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11","Type":"ContainerDied","Data":"a0657c1eabf1fadcba6249c5623a3b599f852140407791dd569eceb7f00ca66d"}
Sep 17 18:30:01.281591 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.281564    2559 scope.go:117] "RemoveContainer" containerID="a0657c1eabf1fadcba6249c5623a3b599f852140407791dd569eceb7f00ca66d"
Sep 17 18:30:01.283237 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.283194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs4w8" event={"ID":"75347877-8dc6-45b1-be39-82372a6459ec","Type":"ContainerStarted","Data":"31a92b39f29137a9f3402d02f0106a13310cc902b4d027918435faf3d618570d"}
Sep 17 18:30:01.284895 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.284853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp" event={"ID":"5588c483-4167-4b8d-8adc-eecfc98fd800","Type":"ContainerStarted","Data":"19d5cae50b26e974d53d22f3b98f5328fca6df2587a059635592ecd5c043812f"}
Sep 17 18:30:01.286399 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.286377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd" event={"ID":"caef9342-c962-42d4-b237-ed36f6d1a603","Type":"ContainerStarted","Data":"416455199b7d9780eda7f27de4cf53610b600bf16d028edd235a14d0f587f4b3"}
Sep 17 18:30:01.290437 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.290398    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v" podStartSLOduration=12.159095113 podStartE2EDuration="19.290386485s" totalImagesPullingTime="7.131291372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.289290514 +0000 UTC m=+43.839514403" watchObservedRunningTime="2026-09-17 18:30:01.290386485 +0000 UTC m=+43.840610369"
Sep 17 18:30:01.310463 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.310420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd" podStartSLOduration=12.144497301 podStartE2EDuration="19.310404904s" totalImagesPullingTime="7.165907603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.310024479 +0000 UTC m=+43.860248367" watchObservedRunningTime="2026-09-17 18:30:01.310404904 +0000 UTC m=+43.860628790"
Sep 17 18:30:01.331967 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.331915    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xs4w8" podStartSLOduration=2.160594621 podStartE2EDuration="9.331895101s" totalImagesPullingTime="7.17130048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.331039426 +0000 UTC m=+43.881263315" watchObservedRunningTime="2026-09-17 18:30:01.331895101 +0000 UTC m=+43.882118991"
Sep 17 18:30:01.358314 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.358265    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx" podStartSLOduration=13.195917829 podStartE2EDuration="20.358244754s" totalImagesPullingTime="7.162326925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.357441029 +0000 UTC m=+43.907664917" watchObservedRunningTime="2026-09-17 18:30:01.358244754 +0000 UTC m=+43.908468642"
Sep 17 18:30:01.380967 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.380913    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-7qgvc" podStartSLOduration=13.320906794 podStartE2EDuration="20.380893493s" totalImagesPullingTime="7.059986699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.379347765 +0000 UTC m=+43.929571652" watchObservedRunningTime="2026-09-17 18:30:01.380893493 +0000 UTC m=+43.931117378"
Sep 17 18:30:01.408211 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.408014    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bt5vp" podStartSLOduration=12.240094941 podStartE2EDuration="19.407994578s" totalImagesPullingTime="7.167899637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.407767084 +0000 UTC m=+43.957990966" watchObservedRunningTime="2026-09-17 18:30:01.407994578 +0000 UTC m=+43.958218466"
Sep 17 18:30:01.443230 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.441530    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fbrlc" podStartSLOduration=13.63736095 podStartE2EDuration="42.441507278s" totalImagesPullingTime="28.804146328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.441042561 +0000 UTC m=+43.991266475" watchObservedRunningTime="2026-09-17 18:30:01.441507278 +0000 UTC m=+43.991731150"
Sep 17 18:30:01.588515 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.588439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:01.676673 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.676637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:01.775283 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.775248    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:01.876690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:01.876555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.072214 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.062328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.249556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.249517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.292132 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.292104    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:30:02.292593 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.292574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/0.log"
Sep 17 18:30:02.292666 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.292618    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc9bafcc-d25f-4e55-8f3d-5eb321bbde11" containerID="ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38" exitCode=255
Sep 17 18:30:02.292762 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.292738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" event={"ID":"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11","Type":"ContainerDied","Data":"ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38"}
Sep 17 18:30:02.292920 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.292796    2559 scope.go:117] "RemoveContainer" containerID="a0657c1eabf1fadcba6249c5623a3b599f852140407791dd569eceb7f00ca66d"
Sep 17 18:30:02.293239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.293073    2559 scope.go:117] "RemoveContainer" containerID="ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38"
Sep 17 18:30:02.293346 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:02.293261    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xcgc8_openshift-console-operator(cc9bafcc-d25f-4e55-8f3d-5eb321bbde11)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" podUID="cc9bafcc-d25f-4e55-8f3d-5eb321bbde11"
Sep 17 18:30:02.571341 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.571251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:30:02.571341 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.571309    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:30:02.677210 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.677176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.804887 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.804854    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gwwfq"]
Sep 17 18:30:02.824918 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.824819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.834717 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.834680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwwfq"]
Sep 17 18:30:02.834860 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.834805    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.838761 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.838731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:02.838919 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.838900    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:02.838988 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.838917    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gvz8\""
Sep 17 18:30:02.887365 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.887335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.887365 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.887368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5505393-5cb7-46ff-93ac-aa8400c4f92d-crio-socket\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.887611 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.887409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgsz\" (UniqueName: \"kubernetes.io/projected/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-api-access-glgsz\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.887611 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.887520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.887713 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.887640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5505393-5cb7-46ff-93ac-aa8400c4f92d-data-volume\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988463 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5505393-5cb7-46ff-93ac-aa8400c4f92d-data-volume\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5505393-5cb7-46ff-93ac-aa8400c4f92d-crio-socket\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgsz\" (UniqueName: \"kubernetes.io/projected/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-api-access-glgsz\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:02.988614    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:02.988660 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:02.988637    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:02.988941 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:02.988691    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.488674271 +0000 UTC m=+46.038898136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:02.988941 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:02.988706    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.488699397 +0000 UTC m=+46.038923262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:02.988941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5505393-5cb7-46ff-93ac-aa8400c4f92d-crio-socket\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:02.988941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:02.988866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5505393-5cb7-46ff-93ac-aa8400c4f92d-data-volume\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:03.000134 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.000111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgsz\" (UniqueName: \"kubernetes.io/projected/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-api-access-glgsz\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:03.211554 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.211475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.297664 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.297636    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:30:03.298153 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.298062    2559 scope.go:117] "RemoveContainer" containerID="ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38"
Sep 17 18:30:03.298264 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:03.298243    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xcgc8_openshift-console-operator(cc9bafcc-d25f-4e55-8f3d-5eb321bbde11)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" podUID="cc9bafcc-d25f-4e55-8f3d-5eb321bbde11"
Sep 17 18:30:03.363862 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.363813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.493329 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.493239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:03.493507 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:03.493349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:03.493507 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:03.493382    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:03.493507 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:03.493459    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:04.493438568 +0000 UTC m=+47.043662433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:03.493507 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:03.493462    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:03.493507 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:03.493506    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:04.493490088 +0000 UTC m=+47.043713956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:04.301170 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.301146    2559 scope.go:117] "RemoveContainer" containerID="ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38"
Sep 17 18:30:04.301528 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:04.301290    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xcgc8_openshift-console-operator(cc9bafcc-d25f-4e55-8f3d-5eb321bbde11)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" podUID="cc9bafcc-d25f-4e55-8f3d-5eb321bbde11"
Sep 17 18:30:04.372740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.372706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.445825 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.445795    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.506711 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.506676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:04.506858 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.506785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:04.506858 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:04.506791    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:04.506932 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:04.506865    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:06.506849041 +0000 UTC m=+49.057072906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:04.506932 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:04.506908    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:04.507007 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:04.506951    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:06.506940658 +0000 UTC m=+49.057164522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:04.883698 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.883667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.934744 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.934716    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lw8h_8fe2d461-8d80-4cb2-86fd-436d591bdafc/dns-node-resolver/0.log"
Sep 17 18:30:04.983639 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:04.983606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:05.855781 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:05.855748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.335794 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:06.335767    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rl9dv_ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6/node-ca/0.log"
Sep 17 18:30:06.524267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:06.524230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:06.524427 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:06.524334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:06.524427 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:06.524357    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.524427 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:06.524422    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.524401899 +0000 UTC m=+53.074625765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.524538 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:06.524467    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.524538 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:06.524516    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.524504162 +0000 UTC m=+53.074728027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.934675 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:06.934642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs4w8_75347877-8dc6-45b1-be39-82372a6459ec/serve-healthcheck-canary/0.log"
Sep 17 18:30:07.544130 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:07.544101    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-gs7jx_c0d6e2ef-a0b6-4517-ae26-54d0803e3599/kube-storage-version-migrator-operator/0.log"
Sep 17 18:30:08.138666 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.138627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:08.139086 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.138794    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:08.139086 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.138815    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d4f867-8jt82: secret "image-registry-tls" not found
Sep 17 18:30:08.139086 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.138897    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls podName:7ad9afcc-5e14-459a-b880-b139113a3d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.138881424 +0000 UTC m=+66.689105293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls") pod "image-registry-7f4d4f867-8jt82" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81") : secret "image-registry-tls" not found
Sep 17 18:30:08.239729 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.239685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:08.239944 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.239762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:08.239944 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.239788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:08.239944 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.239892    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.239869534 +0000 UTC m=+66.790093403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:08.239944 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.239925    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:08.239944 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.239939    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:08.240159 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.239978    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs podName:3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.239966712 +0000 UTC m=+66.790190576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs") pod "router-default-7d4456f768-xpfh4" (UID: "3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4") : secret "router-metrics-certs-default" not found
Sep 17 18:30:08.240159 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.239997    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls podName:53bcfcd2-8b30-48bb-893f-74f7eefe20e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.239984129 +0000 UTC m=+66.790207997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls") pod "dns-default-hskxl" (UID: "53bcfcd2-8b30-48bb-893f-74f7eefe20e8") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:08.240159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.240017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:30:08.240159 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.240115    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:08.240159 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.240155    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert podName:396a48fe-1cfd-478c-ae9a-3e2eb49d5017 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.240145131 +0000 UTC m=+66.790368996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jx4ts" (UID: "396a48fe-1cfd-478c-ae9a-3e2eb49d5017") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:08.340892 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.340853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:30:08.341092 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.340920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:30:08.341092 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.340952    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:08.341092 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.341002    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls podName:d0ee7e92-446b-4fd2-817f-363399f71b8c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.340989268 +0000 UTC m=+66.891213134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n9xrd" (UID: "d0ee7e92-446b-4fd2-817f-363399f71b8c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:08.341092 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.341069    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:08.341312 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:08.341121    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls podName:2c566714-79e3-4017-874b-8563302c1baa nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.341105925 +0000 UTC m=+66.891329790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-pdclq" (UID: "2c566714-79e3-4017-874b-8563302c1baa") : secret "samples-operator-tls" not found
Sep 17 18:30:08.341667 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.341639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.435243 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.435159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.536166 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.536131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.663178 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.663147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.823661 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.823626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.895645 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:08.895617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.110937 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:09.110857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.728716 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:09.728684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.828731 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:09.828700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.893937 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:09.893904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.227784 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.227754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.425880 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.425845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.526362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.526327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.561630 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.561604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:10.561746 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.561713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:10.561782 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:10.561751    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.561819 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:30:10.561812    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls podName:c5505393-5cb7-46ff-93ac-aa8400c4f92d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:18.561797543 +0000 UTC m=+61.112021408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwwfq" (UID: "c5505393-5cb7-46ff-93ac-aa8400c4f92d") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.562279 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.562261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5505393-5cb7-46ff-93ac-aa8400c4f92d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:10.817634 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:10.817552    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:11.620187 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:11.620155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:11.941916 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:11.941822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.045644 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:12.045614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.220174 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:12.220099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.408349 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:12.408318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.811503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:12.811470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.887806 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:12.887775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.609254 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:13.609223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.806145 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:13.806110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.903299 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:13.903222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.003977 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:14.003946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.203692 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:14.203613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.398694 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:14.398662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.614242 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:14.614200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.804483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:14.804454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.404376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:15.404342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.812568 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:15.812539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.199684 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:16.199593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.399845 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:16.399796    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.802408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:16.802375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.389940 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:17.389907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.615125 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:17.615089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.815502 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:17.815475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.214927 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.214821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.224555 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.224526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v52kc"
Sep 17 18:30:18.402072 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.402035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.629995 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.629946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:18.632290 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.632268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5505393-5cb7-46ff-93ac-aa8400c4f92d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwwfq\" (UID: \"c5505393-5cb7-46ff-93ac-aa8400c4f92d\") " pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:18.746890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.746858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gvz8\""
Sep 17 18:30:18.754268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.754246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwwfq"
Sep 17 18:30:18.837106 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.837074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.936101 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:18.936024    2559 scope.go:117] "RemoveContainer" containerID="ef01c9b05f645bb86b468bb3e0787b9dba16c8aa0c9d34dd524b85fea2644e38"
Sep 17 18:30:19.029535 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.029505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwwfq"]
Sep 17 18:30:19.033427 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:19.033402    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5505393_5cb7_46ff_93ac_aa8400c4f92d.slice/crio-e31f3e909dfd734c33306343b79a91e5f1df56a25ca9d6cc4d51082048836c11 WatchSource:0}: Error finding container e31f3e909dfd734c33306343b79a91e5f1df56a25ca9d6cc4d51082048836c11: Status 404 returned error can't find the container with id e31f3e909dfd734c33306343b79a91e5f1df56a25ca9d6cc4d51082048836c11
Sep 17 18:30:19.207388 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.207304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.349304 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.349280    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:30:19.349470 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.349400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" event={"ID":"cc9bafcc-d25f-4e55-8f3d-5eb321bbde11","Type":"ContainerStarted","Data":"a919e7adbae63c8dcb5f7f751dea0c06d389bdadcc4973cca4aa7aa345470a64"}
Sep 17 18:30:19.349842 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.349766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:30:19.351072 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.351038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwwfq" event={"ID":"c5505393-5cb7-46ff-93ac-aa8400c4f92d","Type":"ContainerStarted","Data":"d5fa34c04773bdcee6f7fecddce682f69a828716f4cd5e1f5d7456863e2a1f55"}
Sep 17 18:30:19.351185 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.351091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwwfq" event={"ID":"c5505393-5cb7-46ff-93ac-aa8400c4f92d","Type":"ContainerStarted","Data":"e31f3e909dfd734c33306343b79a91e5f1df56a25ca9d6cc4d51082048836c11"}
Sep 17 18:30:19.370614 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.370571    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8" podStartSLOduration=30.914933157 podStartE2EDuration="38.370558401s" totalImagesPullingTime="7.455625244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.369685123 +0000 UTC m=+61.919909022" watchObservedRunningTime="2026-09-17 18:30:19.370558401 +0000 UTC m=+61.920782287"
Sep 17 18:30:19.446000 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.445972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-xcgc8"
Sep 17 18:30:19.624969 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.624918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.726396 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.726350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.833030 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.832985    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.950036 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:19.949958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.115400 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.115367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.301798 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.301741    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.356778 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.356738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwwfq" event={"ID":"c5505393-5cb7-46ff-93ac-aa8400c4f92d","Type":"ContainerStarted","Data":"e530f751d31409abe35162a1feaa556ab8d8e254574f90e558fc3e37b1758a85"}
Sep 17 18:30:20.363273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.363244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.786583 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.786544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.845277 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.845242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.935579 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:20.935542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.002062 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.002019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.312316 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.312282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.361485 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.361411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwwfq" event={"ID":"c5505393-5cb7-46ff-93ac-aa8400c4f92d","Type":"ContainerStarted","Data":"83b77dc48134c49ff4778f4ce72b61bce327c0dc5a9d082558da85098617f6ca"}
Sep 17 18:30:21.384693 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.382251    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gwwfq" podStartSLOduration=17.446297969 podStartE2EDuration="19.382230936s" totalImagesPullingTime="1.935932967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:21.381297588 +0000 UTC m=+63.931521481" watchObservedRunningTime="2026-09-17 18:30:21.382230936 +0000 UTC m=+63.932454824"
Sep 17 18:30:21.513294 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.513264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.733463 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:21.733375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.779965 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:22.779930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.147883 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.147780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.259765 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.259730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.576874 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.576843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:30:23.579648 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.579627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:23.589962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.589937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b3f519-272f-4c6a-8cd1-f78ed4884548-metrics-certs\") pod \"network-metrics-daemon-mtf79\" (UID: \"38b3f519-272f-4c6a-8cd1-f78ed4884548\") " pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:30:23.658262 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.658232    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzhw6\""
Sep 17 18:30:23.666502 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.666474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtf79"
Sep 17 18:30:23.678204 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.678173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:30:23.680737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.680720    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:23.691041 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.691014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/923b92fd-9415-43ec-ab08-465f6cd44407-original-pull-secret\") pod \"global-pull-secret-syncer-rf6t6\" (UID: \"923b92fd-9415-43ec-ab08-465f6cd44407\") " pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:30:23.778593 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.778559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:30:23.781077 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.781053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gqh\" (UniqueName: \"kubernetes.io/projected/695d5ef7-dc4e-41b4-9aab-a9500c6dfc97-kube-api-access-k5gqh\") pod \"network-check-target-42qln\" (UID: \"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97\") " pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:30:23.965262 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.965187    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtf79"]
Sep 17 18:30:23.966931 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.966914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rf6t6"
Sep 17 18:30:23.968698 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:23.968671    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b3f519_272f_4c6a_8cd1_f78ed4884548.slice/crio-0003cc41cfc00170357da735a907b9651cb9cc0db5c65b4f6c4f1094bd13ed49 WatchSource:0}: Error finding container 0003cc41cfc00170357da735a907b9651cb9cc0db5c65b4f6c4f1094bd13ed49: Status 404 returned error can't find the container with id 0003cc41cfc00170357da735a907b9651cb9cc0db5c65b4f6c4f1094bd13ed49
Sep 17 18:30:23.977791 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.977761    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rcjbl\""
Sep 17 18:30:23.986081 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:23.986058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:30:24.182138 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.182095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:24.184540 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.184514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"image-registry-7f4d4f867-8jt82\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") " pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:24.243112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.243036    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rf6t6"]
Sep 17 18:30:24.245821 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.245795    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923b92fd_9415_43ec_ab08_465f6cd44407.slice/crio-05995270517d40f7fb07ed0101cbf83d0f6bc16fe816412c218a3227c7b0c126 WatchSource:0}: Error finding container 05995270517d40f7fb07ed0101cbf83d0f6bc16fe816412c218a3227c7b0c126: Status 404 returned error can't find the container with id 05995270517d40f7fb07ed0101cbf83d0f6bc16fe816412c218a3227c7b0c126
Sep 17 18:30:24.281503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.281469    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42qln"]
Sep 17 18:30:24.283376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.283273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:30:24.283376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.283304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:24.283376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.283335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:24.283561 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.283385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:24.284063 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.284040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-service-ca-bundle\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:24.285485 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.285460    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695d5ef7_dc4e_41b4_9aab_a9500c6dfc97.slice/crio-fd019ffcb43b3bdab47cb90b1adadb9d44c0c22cb64dd1d28b5b36609c1eb658 WatchSource:0}: Error finding container fd019ffcb43b3bdab47cb90b1adadb9d44c0c22cb64dd1d28b5b36609c1eb658: Status 404 returned error can't find the container with id fd019ffcb43b3bdab47cb90b1adadb9d44c0c22cb64dd1d28b5b36609c1eb658
Sep 17 18:30:24.285914 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.285891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/396a48fe-1cfd-478c-ae9a-3e2eb49d5017-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jx4ts\" (UID: \"396a48fe-1cfd-478c-ae9a-3e2eb49d5017\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:30:24.286049 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.286033    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53bcfcd2-8b30-48bb-893f-74f7eefe20e8-metrics-tls\") pod \"dns-default-hskxl\" (UID: \"53bcfcd2-8b30-48bb-893f-74f7eefe20e8\") " pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:24.286105 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.286093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4-metrics-certs\") pod \"router-default-7d4456f768-xpfh4\" (UID: \"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4\") " pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:24.374568 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.373962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42qln" event={"ID":"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97","Type":"ContainerStarted","Data":"e48bcd1e167bc3c0e20c8bb26e07e4f7f8263321c10751754fa66744383f5d8d"}
Sep 17 18:30:24.374568 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.374024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42qln" event={"ID":"695d5ef7-dc4e-41b4-9aab-a9500c6dfc97","Type":"ContainerStarted","Data":"fd019ffcb43b3bdab47cb90b1adadb9d44c0c22cb64dd1d28b5b36609c1eb658"}
Sep 17 18:30:24.374805 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.374662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:30:24.376426 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.376393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rf6t6" event={"ID":"923b92fd-9415-43ec-ab08-465f6cd44407","Type":"ContainerStarted","Data":"05995270517d40f7fb07ed0101cbf83d0f6bc16fe816412c218a3227c7b0c126"}
Sep 17 18:30:24.377681 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.377656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtf79" event={"ID":"38b3f519-272f-4c6a-8cd1-f78ed4884548","Type":"ContainerStarted","Data":"0003cc41cfc00170357da735a907b9651cb9cc0db5c65b4f6c4f1094bd13ed49"}
Sep 17 18:30:24.384136 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.384120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:30:24.384210 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.384178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:30:24.386479 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.386453    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0ee7e92-446b-4fd2-817f-363399f71b8c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n9xrd\" (UID: \"d0ee7e92-446b-4fd2-817f-363399f71b8c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:30:24.386572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.386538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c566714-79e3-4017-874b-8563302c1baa-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-pdclq\" (UID: \"2c566714-79e3-4017-874b-8563302c1baa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:30:24.390204 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.390157    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-42qln" podStartSLOduration=65.390141473 podStartE2EDuration="1m5.390141473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:24.389907486 +0000 UTC m=+66.940131374" watchObservedRunningTime="2026-09-17 18:30:24.390141473 +0000 UTC m=+66.940365354"
Sep 17 18:30:24.394117 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.394100    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-26p5d\""
Sep 17 18:30:24.402160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.402143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:24.421442 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.421420    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m49wr\""
Sep 17 18:30:24.429539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.429516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"
Sep 17 18:30:24.470706 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.470623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wk4f4\""
Sep 17 18:30:24.479233 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.479193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:24.489962 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.489942    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpkjn\""
Sep 17 18:30:24.498205 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.498178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:24.535589 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.535302    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n9qvt\""
Sep 17 18:30:24.543280 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.543033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"
Sep 17 18:30:24.568784 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.568757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dm7mx\""
Sep 17 18:30:24.577074 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.577040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"
Sep 17 18:30:24.682733 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.682666    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:30:24.687148 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.687122    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad9afcc_5e14_459a_b880_b139113a3d81.slice/crio-089e669a9a74ac65ee152119b3c65ca5198fd0d62f89d97c573ae46405bc6245 WatchSource:0}: Error finding container 089e669a9a74ac65ee152119b3c65ca5198fd0d62f89d97c573ae46405bc6245: Status 404 returned error can't find the container with id 089e669a9a74ac65ee152119b3c65ca5198fd0d62f89d97c573ae46405bc6245
Sep 17 18:30:24.700702 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.700676    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts"]
Sep 17 18:30:24.704203 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.704130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396a48fe_1cfd_478c_ae9a_3e2eb49d5017.slice/crio-057ffb50ad7d88d08b1d83f3f97575962072068104d69f2ae846f497182cd1cd WatchSource:0}: Error finding container 057ffb50ad7d88d08b1d83f3f97575962072068104d69f2ae846f497182cd1cd: Status 404 returned error can't find the container with id 057ffb50ad7d88d08b1d83f3f97575962072068104d69f2ae846f497182cd1cd
Sep 17 18:30:24.788982 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.788952    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d4456f768-xpfh4"]
Sep 17 18:30:24.794857 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.794806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hskxl"]
Sep 17 18:30:24.796477 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.796448    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6b2e61_d20e_4c70_98b4_03d8f5b84cd4.slice/crio-57ba085ea39dcf099fd6c15843eecf8d10e6a374bf21d380aaed4081cd130dc7 WatchSource:0}: Error finding container 57ba085ea39dcf099fd6c15843eecf8d10e6a374bf21d380aaed4081cd130dc7: Status 404 returned error can't find the container with id 57ba085ea39dcf099fd6c15843eecf8d10e6a374bf21d380aaed4081cd130dc7
Sep 17 18:30:24.797052 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.797024    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bcfcd2_8b30_48bb_893f_74f7eefe20e8.slice/crio-74737b2d00e34d76ec11bf918925d17f74069ba7b8565c31510a8f5048314bcb WatchSource:0}: Error finding container 74737b2d00e34d76ec11bf918925d17f74069ba7b8565c31510a8f5048314bcb: Status 404 returned error can't find the container with id 74737b2d00e34d76ec11bf918925d17f74069ba7b8565c31510a8f5048314bcb
Sep 17 18:30:24.838580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.838375    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq"]
Sep 17 18:30:24.867931 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:24.867896    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd"]
Sep 17 18:30:24.873104 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:24.873070    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ee7e92_446b_4fd2_817f_363399f71b8c.slice/crio-9e5f51b99cbe8471de56abed8d73414e85930ad81764e39bcf3e59fb6b0a3629 WatchSource:0}: Error finding container 9e5f51b99cbe8471de56abed8d73414e85930ad81764e39bcf3e59fb6b0a3629: Status 404 returned error can't find the container with id 9e5f51b99cbe8471de56abed8d73414e85930ad81764e39bcf3e59fb6b0a3629
Sep 17 18:30:25.385883 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.385026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" event={"ID":"7ad9afcc-5e14-459a-b880-b139113a3d81","Type":"ContainerStarted","Data":"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"}
Sep 17 18:30:25.385883 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.385093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" event={"ID":"7ad9afcc-5e14-459a-b880-b139113a3d81","Type":"ContainerStarted","Data":"089e669a9a74ac65ee152119b3c65ca5198fd0d62f89d97c573ae46405bc6245"}
Sep 17 18:30:25.386143 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.385972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:25.388113 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.388055    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq" event={"ID":"2c566714-79e3-4017-874b-8563302c1baa","Type":"ContainerStarted","Data":"40171ce690cf7ac278b508218c90c4f713d29f3797e121842205cb81bc2ffa33"}
Sep 17 18:30:25.395338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.395236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hskxl" event={"ID":"53bcfcd2-8b30-48bb-893f-74f7eefe20e8","Type":"ContainerStarted","Data":"74737b2d00e34d76ec11bf918925d17f74069ba7b8565c31510a8f5048314bcb"}
Sep 17 18:30:25.398389 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.398353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d4456f768-xpfh4" event={"ID":"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4","Type":"ContainerStarted","Data":"264982e92b5a177cd65a69f74bd25a2683c9544697b4a33ec9d064f8d297948d"}
Sep 17 18:30:25.398511 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.398401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d4456f768-xpfh4" event={"ID":"3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4","Type":"ContainerStarted","Data":"57ba085ea39dcf099fd6c15843eecf8d10e6a374bf21d380aaed4081cd130dc7"}
Sep 17 18:30:25.400733 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.400617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts" event={"ID":"396a48fe-1cfd-478c-ae9a-3e2eb49d5017","Type":"ContainerStarted","Data":"057ffb50ad7d88d08b1d83f3f97575962072068104d69f2ae846f497182cd1cd"}
Sep 17 18:30:25.402041 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.402005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd" event={"ID":"d0ee7e92-446b-4fd2-817f-363399f71b8c","Type":"ContainerStarted","Data":"9e5f51b99cbe8471de56abed8d73414e85930ad81764e39bcf3e59fb6b0a3629"}
Sep 17 18:30:25.413323 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.413102    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" podStartSLOduration=66.413065926 podStartE2EDuration="1m6.413065926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:25.411914383 +0000 UTC m=+67.962138274" watchObservedRunningTime="2026-09-17 18:30:25.413065926 +0000 UTC m=+67.963289812"
Sep 17 18:30:25.432505 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.432401    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d4456f768-xpfh4" podStartSLOduration=44.432381998 podStartE2EDuration="44.432381998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:25.431438015 +0000 UTC m=+67.981661915" watchObservedRunningTime="2026-09-17 18:30:25.432381998 +0000 UTC m=+67.982605890"
Sep 17 18:30:25.480301 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.480005    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:25.483355 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:25.483164    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:26.409629 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.408805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtf79" event={"ID":"38b3f519-272f-4c6a-8cd1-f78ed4884548","Type":"ContainerStarted","Data":"4f01521d342f7f7f07d5ba6af48524c9ff978102f2006d77fb128d29efba550d"}
Sep 17 18:30:26.409629 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.408892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtf79" event={"ID":"38b3f519-272f-4c6a-8cd1-f78ed4884548","Type":"ContainerStarted","Data":"6bbd72cca78bc3c58d179208ddced3235da99a2f84e84245d4740ae30d7d61b7"}
Sep 17 18:30:26.412721 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.412235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts" event={"ID":"396a48fe-1cfd-478c-ae9a-3e2eb49d5017","Type":"ContainerStarted","Data":"d0af30ad88c111e33dc00fd7ac91dee39b1ad554f597d2b6b0bdb76427b1693e"}
Sep 17 18:30:26.412721 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.412291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:26.414372 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.414325    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d4456f768-xpfh4"
Sep 17 18:30:26.427643 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.427591    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mtf79" podStartSLOduration=66.954053791 podStartE2EDuration="1m8.427575089s" totalImagesPullingTime="1.473521298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.425991199 +0000 UTC m=+68.976215098" watchObservedRunningTime="2026-09-17 18:30:26.427575089 +0000 UTC m=+68.977798976"
Sep 17 18:30:26.443763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:26.442506    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jx4ts" podStartSLOduration=44.152504548 podStartE2EDuration="45.442488733s" totalImagesPullingTime="1.289984185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.44085584 +0000 UTC m=+68.991079730" watchObservedRunningTime="2026-09-17 18:30:26.442488733 +0000 UTC m=+68.992712620"
Sep 17 18:30:27.311975 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:27.311941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.474533 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:27.474444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.631681 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:27.631639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.756041 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:27.756008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.034882 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.034842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.140219 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.140170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.250909 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.250875    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.427975 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.427937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rf6t6" event={"ID":"923b92fd-9415-43ec-ab08-465f6cd44407","Type":"ContainerStarted","Data":"ac34e6fa7628f272baa27c30d421be82832d29a7c167efa5c60474340fcab7b2"}
Sep 17 18:30:30.429663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.429639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd" event={"ID":"d0ee7e92-446b-4fd2-817f-363399f71b8c","Type":"ContainerStarted","Data":"21da274aebbc8df900bc52243baa2d17936ab296d61db666a4a7d43324e7fb46"}
Sep 17 18:30:30.431556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.431533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq" event={"ID":"2c566714-79e3-4017-874b-8563302c1baa","Type":"ContainerStarted","Data":"eb7e714266c56ee1471265321d445dbd69dd2f8d4c144f1edb091178cbc8ea47"}
Sep 17 18:30:30.431663 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.431573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq" event={"ID":"2c566714-79e3-4017-874b-8563302c1baa","Type":"ContainerStarted","Data":"577a3693764b6b6c2503212e2eb7535bf8862efe334d0aead9ff9e8d487dda49"}
Sep 17 18:30:30.433180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.433158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hskxl" event={"ID":"53bcfcd2-8b30-48bb-893f-74f7eefe20e8","Type":"ContainerStarted","Data":"bceb72c05ac7ab4277adbdeba93d5269f0d726486268b19896d86838f09b8e17"}
Sep 17 18:30:30.433257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.433199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hskxl" event={"ID":"53bcfcd2-8b30-48bb-893f-74f7eefe20e8","Type":"ContainerStarted","Data":"acd9358b6c72d0527a4374a7749902055f93a6d0c92548c64ea31c8454db6d1b"}
Sep 17 18:30:30.433257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.433237    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:30.461222 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.461182    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rf6t6" podStartSLOduration=66.122109431 podStartE2EDuration="1m11.461168659s" totalImagesPullingTime="5.339059228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:30.460234193 +0000 UTC m=+73.010458079" watchObservedRunningTime="2026-09-17 18:30:30.461168659 +0000 UTC m=+73.011392540"
Sep 17 18:30:30.499776 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.499743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.511515 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.511420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hskxl" podStartSLOduration=33.734965152 podStartE2EDuration="38.511405819s" totalImagesPullingTime="4.776440667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:30.509378134 +0000 UTC m=+73.059602022" watchObservedRunningTime="2026-09-17 18:30:30.511405819 +0000 UTC m=+73.061629705"
Sep 17 18:30:30.511849 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.511808    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-pdclq" podStartSLOduration=43.835079169 podStartE2EDuration="48.51180111s" totalImagesPullingTime="4.676721941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:30.482199947 +0000 UTC m=+73.032423858" watchObservedRunningTime="2026-09-17 18:30:30.51180111 +0000 UTC m=+73.062025044"
Sep 17 18:30:30.531630 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.531588    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n9xrd" podStartSLOduration=43.831732656 podStartE2EDuration="48.531579503s" totalImagesPullingTime="4.699846847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:30.53120218 +0000 UTC m=+73.081426066" watchObservedRunningTime="2026-09-17 18:30:30.531579503 +0000 UTC m=+73.081803389"
Sep 17 18:30:30.602867 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.602800    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.703966 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:30.703927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.851938 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:32.851908    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.952334 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:32.952295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.291453 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:36.291423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.393676 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:36.393631    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.473728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:36.473692    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:37.352011 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:37.351973    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:37.430464 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:37.430416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.104714 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:38.104676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.206647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:38.206611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.816455 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:38.816414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.271097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.270935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.371399 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.371368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.439857 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.438880    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hskxl"
Sep 17 18:30:40.489375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.489342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.589756 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.589685    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.793539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.793510    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.916332 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.916250    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ssxzd"]
Sep 17 18:30:40.918824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.918807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:40.936787 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.936766    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:40.939044 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.939027    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:40.940308 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.940293    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-24224\""
Sep 17 18:30:40.941387 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.941371    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:40.943672 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:40.943653    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:41.002954 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.002919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.021349 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-textfile\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021349 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021592 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-tls\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021592 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-metrics-client-ca\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021592 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-root\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021592 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-wtmp\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021769 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74lm\" (UniqueName: \"kubernetes.io/projected/abeac62d-87c5-4547-9768-ac33b6b70dad-kube-api-access-g74lm\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021769 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-sys\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.021769 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.021665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.090103 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.090066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.122905 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.122877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74lm\" (UniqueName: \"kubernetes.io/projected/abeac62d-87c5-4547-9768-ac33b6b70dad-kube-api-access-g74lm\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.122916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-sys\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.122942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.122983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-textfile\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-sys\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-tls\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123359 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-metrics-client-ca\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123359 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-root\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123359 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-wtmp\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123359 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-wtmp\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123528 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-textfile\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123528 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/abeac62d-87c5-4547-9768-ac33b6b70dad-root\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123636 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.123879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.123854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abeac62d-87c5-4547-9768-ac33b6b70dad-metrics-client-ca\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.125579 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.125559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.125900 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.125880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/abeac62d-87c5-4547-9768-ac33b6b70dad-node-exporter-tls\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.132814 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.132786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74lm\" (UniqueName: \"kubernetes.io/projected/abeac62d-87c5-4547-9768-ac33b6b70dad-kube-api-access-g74lm\") pod \"node-exporter-ssxzd\" (UID: \"abeac62d-87c5-4547-9768-ac33b6b70dad\") " pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.227704 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.227621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssxzd"
Sep 17 18:30:41.237602 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:41.237574    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeac62d_87c5_4547_9768_ac33b6b70dad.slice/crio-58a5d009370942dc94149fe70bba72c3499d4bfbee96525a57ca1d21637f9143 WatchSource:0}: Error finding container 58a5d009370942dc94149fe70bba72c3499d4bfbee96525a57ca1d21637f9143: Status 404 returned error can't find the container with id 58a5d009370942dc94149fe70bba72c3499d4bfbee96525a57ca1d21637f9143
Sep 17 18:30:41.344938 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.344900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.473846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.473798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssxzd" event={"ID":"abeac62d-87c5-4547-9768-ac33b6b70dad","Type":"ContainerStarted","Data":"58a5d009370942dc94149fe70bba72c3499d4bfbee96525a57ca1d21637f9143"}
Sep 17 18:30:41.681856 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.681806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.780568 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.780534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.865027 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.864989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.991048 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:41.990966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:42.479106 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:42.479074    2559 generic.go:356] "Generic (PLEG): container finished" podID="abeac62d-87c5-4547-9768-ac33b6b70dad" containerID="33afd2df8d226e4048287e76a58bd8d5dc1c8ca04735b88ba88dd7b60dd3737c" exitCode=0
Sep 17 18:30:42.479479 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:42.479156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssxzd" event={"ID":"abeac62d-87c5-4547-9768-ac33b6b70dad","Type":"ContainerDied","Data":"33afd2df8d226e4048287e76a58bd8d5dc1c8ca04735b88ba88dd7b60dd3737c"}
Sep 17 18:30:43.485560 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.485517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssxzd" event={"ID":"abeac62d-87c5-4547-9768-ac33b6b70dad","Type":"ContainerStarted","Data":"30a526919faa8378b526f363fbefe8e56e1517427ec435d2d7e10222df23d71c"}
Sep 17 18:30:43.485956 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.485577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssxzd" event={"ID":"abeac62d-87c5-4547-9768-ac33b6b70dad","Type":"ContainerStarted","Data":"b676d96e693dc61f63b9dc4586abdf224070f4f3aeed86c3ad2e5756d790d3c7"}
Sep 17 18:30:43.508242 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.508189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ssxzd" podStartSLOduration=2.444958512 podStartE2EDuration="3.50817364s" totalImagesPullingTime="1.001257016s" totalInitContainerRuntime="61.958112ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:43.507357509 +0000 UTC m=+86.057581395" watchObservedRunningTime="2026-09-17 18:30:43.50817364 +0000 UTC m=+86.058397560"
Sep 17 18:30:43.679747 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.679719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.785390 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.785354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.861271 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.861232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.941608 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:43.941578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.406682 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:44.406645    2559 patch_prober.go:28] interesting pod/image-registry-7f4d4f867-8jt82 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:30:44.406863 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:44.406705    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:30:45.402432 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.402399    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-656f8dbc68-lrlwz"]
Sep 17 18:30:45.404533 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.404515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.409874 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.409854    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:30:45.411099 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.411078    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zw4m8\""
Sep 17 18:30:45.411204 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.411188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:30:45.411323 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.411308    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:30:45.411392 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.411378    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:30:45.411525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.411510    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dnvpnus8njc7c\""
Sep 17 18:30:45.419544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.419521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-656f8dbc68-lrlwz"]
Sep 17 18:30:45.554867 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.554807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-client-certs\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.554880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-client-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.554949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a617db0e-a75a-475b-a60a-e9bee4e463e2-audit-log\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.554998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-metrics-server-audit-profiles\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.555039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgprp\" (UniqueName: \"kubernetes.io/projected/a617db0e-a75a-475b-a60a-e9bee4e463e2-kube-api-access-kgprp\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.555062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-tls\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.555239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.555079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.562102 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.562064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.655986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.655901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-client-certs\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.655986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.655937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-client-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.655986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.655977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a617db0e-a75a-475b-a60a-e9bee4e463e2-audit-log\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.655996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-metrics-server-audit-profiles\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.656012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgprp\" (UniqueName: \"kubernetes.io/projected/a617db0e-a75a-475b-a60a-e9bee4e463e2-kube-api-access-kgprp\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.656030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-tls\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656257 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.656050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656450 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.656424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a617db0e-a75a-475b-a60a-e9bee4e463e2-audit-log\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.656937 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.656907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.657121 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.657100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a617db0e-a75a-475b-a60a-e9bee4e463e2-metrics-server-audit-profiles\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.658572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.658546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-client-certs\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.658572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.658561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-client-ca-bundle\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.658704 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.658586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a617db0e-a75a-475b-a60a-e9bee4e463e2-secret-metrics-server-tls\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.674992 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.674969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgprp\" (UniqueName: \"kubernetes.io/projected/a617db0e-a75a-475b-a60a-e9bee4e463e2-kube-api-access-kgprp\") pod \"metrics-server-656f8dbc68-lrlwz\" (UID: \"a617db0e-a75a-475b-a60a-e9bee4e463e2\") " pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.713235 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.713203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:30:45.762095 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.761946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.855518 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.855473    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.997549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:45.997522    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-656f8dbc68-lrlwz"]
Sep 17 18:30:45.998458 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:45.998429    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda617db0e_a75a_475b_a60a_e9bee4e463e2.slice/crio-8472dd1c30c3eb3c4961d2db7a0fb0adb4d6cea65ccfabc96fa928180920b946 WatchSource:0}: Error finding container 8472dd1c30c3eb3c4961d2db7a0fb0adb4d6cea65ccfabc96fa928180920b946: Status 404 returned error can't find the container with id 8472dd1c30c3eb3c4961d2db7a0fb0adb4d6cea65ccfabc96fa928180920b946
Sep 17 18:30:46.498659 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:46.498624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz" event={"ID":"a617db0e-a75a-475b-a60a-e9bee4e463e2","Type":"ContainerStarted","Data":"8472dd1c30c3eb3c4961d2db7a0fb0adb4d6cea65ccfabc96fa928180920b946"}
Sep 17 18:30:47.030129 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.030096    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:30:47.032595 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.032572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.035564 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.035544    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:30:47.037986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.037964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:30:47.038097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.038001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:30:47.038097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.038001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:30:47.038097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.038001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:30:47.038259 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.037962    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:30:47.040805 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.040786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.045185 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.045162    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vpw9j\""
Sep 17 18:30:47.045295 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.045245    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:30:47.045295 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.045253    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:30:47.046282 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.046027    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:30:47.046282 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.046068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:30:47.046282 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.046131    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5hsd27l5ep16f\""
Sep 17 18:30:47.046518 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.046366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:30:47.046518 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.046401    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:30:47.048081 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.048061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:30:47.053359 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.053342    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:30:47.084934 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.084905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:30:47.168553 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.168728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.168728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.168728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.168728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.168728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwh4\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.168999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.169531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.169273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270092 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270263 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270263 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwh4\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270500 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270557 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270557 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.270890 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.271180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.271180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.271180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.271180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.270962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.271396 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.271237    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.272378 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.271471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.272378 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.272119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.273600 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.273533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.273974 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.273951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.274084 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.274040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.274922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.274555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.274922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.274618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.274922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.274671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.274922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.274879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.275375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.275347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.275673 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.275650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.276238 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.276216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.276852 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.276813    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.277549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.277478    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.277881 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.277863    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.278539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.278471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.281435 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.281386    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwh4\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4\") pod \"prometheus-k8s-0\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.344427 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.344388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:47.420820 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.420794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:30:47.504532 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.504499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz" event={"ID":"a617db0e-a75a-475b-a60a-e9bee4e463e2","Type":"ContainerStarted","Data":"725f9f5a78bf10a6540fa2389b53b2f1ad1188e03a373a7fff37464058fa756e"}
Sep 17 18:30:47.528859 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.528793    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz" podStartSLOduration=1.102520161 podStartE2EDuration="2.528776216s" totalImagesPullingTime="1.426256055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:47.528204094 +0000 UTC m=+90.078427983" watchObservedRunningTime="2026-09-17 18:30:47.528776216 +0000 UTC m=+90.079000120"
Sep 17 18:30:47.640931 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.640844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.685916 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.685891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:30:47.688311 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:30:47.688280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31b77f2_1119_427e_a057_07a8cc2d399c.slice/crio-71aaaea9827a293c77724c0ea7634411cdc9b63ea8b56e617b5946243c68b79b WatchSource:0}: Error finding container 71aaaea9827a293c77724c0ea7634411cdc9b63ea8b56e617b5946243c68b79b: Status 404 returned error can't find the container with id 71aaaea9827a293c77724c0ea7634411cdc9b63ea8b56e617b5946243c68b79b
Sep 17 18:30:47.741736 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:47.741695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.509610 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:48.509574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"71aaaea9827a293c77724c0ea7634411cdc9b63ea8b56e617b5946243c68b79b"}
Sep 17 18:30:49.514510 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:49.514476    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202" exitCode=0
Sep 17 18:30:49.514961 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:49.514562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"}
Sep 17 18:30:49.593320 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:49.593291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.669215 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:49.669186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.946322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:49.946242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:50.020068 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:50.020030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.141233 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:51.141201    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:30:51.263536 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:51.263485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.391208 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:51.391175    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.368472 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:52.368439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.518057 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:52.518000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.531201 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:52.531171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"}
Sep 17 18:30:52.531338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:52.531236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"}
Sep 17 18:30:53.022528 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:53.022491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.105555 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:53.105523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.546196 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:55.546155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"}
Sep 17 18:30:55.546674 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:55.546231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"}
Sep 17 18:30:55.546674 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:55.546270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"}
Sep 17 18:30:55.546674 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:55.546315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerStarted","Data":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"}
Sep 17 18:30:55.577644 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:55.577589    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.556133468 podStartE2EDuration="8.577572776s" totalImagesPullingTime="6.002747965s" totalInitContainerRuntime="18.691343ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:55.575816707 +0000 UTC m=+98.126040615" watchObservedRunningTime="2026-09-17 18:30:55.577572776 +0000 UTC m=+98.127796662"
Sep 17 18:30:56.198307 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:56.198270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.305611 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:56.305537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.415036 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:56.415001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-42qln"
Sep 17 18:30:57.344923 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:57.344887    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:30:58.570408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:58.570370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:58.671670 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:30:58.671633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:01.672700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:01.672661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:01.784081 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:01.784050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:01.879765 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:01.879727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:02.140681 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:02.140650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:02.237267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:02.237230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:05.714091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:05.714053    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:31:05.714525 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:05.714126    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:31:08.106198 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:08.106160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:08.203823 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:08.203789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:11.609007 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:11.608974    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0d6e2ef-a0b6-4517-ae26-54d0803e3599" containerID="e01f08d8cfb8296a2e544c1340e01f4a90e28d18e15661ea26432fe30a66ac3f" exitCode=0
Sep 17 18:31:11.609395 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:11.609027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx" event={"ID":"c0d6e2ef-a0b6-4517-ae26-54d0803e3599","Type":"ContainerDied","Data":"e01f08d8cfb8296a2e544c1340e01f4a90e28d18e15661ea26432fe30a66ac3f"}
Sep 17 18:31:11.609395 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:11.609317    2559 scope.go:117] "RemoveContainer" containerID="e01f08d8cfb8296a2e544c1340e01f4a90e28d18e15661ea26432fe30a66ac3f"
Sep 17 18:31:12.617506 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:12.617471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gs7jx" event={"ID":"c0d6e2ef-a0b6-4517-ae26-54d0803e3599","Type":"ContainerStarted","Data":"5b3face436892f765c870d2146ae5f862e54e0a41af4403d3404a5b23e446d18"}
Sep 17 18:31:16.162852 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.162770    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerName="registry" containerID="cri-o://1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941" gracePeriod=30
Sep 17 18:31:16.400921 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.400898    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:31:16.536720 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536688    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.536926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536729    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.536926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.536926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536797    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.536926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.536926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536874    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.537201 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.536964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.537201 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.537014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfdbl\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl\") pod \"7ad9afcc-5e14-459a-b880-b139113a3d81\" (UID: \"7ad9afcc-5e14-459a-b880-b139113a3d81\") "
Sep 17 18:31:16.537555 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.537382    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:16.537679 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.537601    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:16.539622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.539589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:16.539750 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.539622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:16.539750 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.539660    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl" (OuterVolumeSpecName: "kube-api-access-bfdbl") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "kube-api-access-bfdbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:16.539750 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.539701    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:16.539899 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.539879    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:16.545249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.545207    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7ad9afcc-5e14-459a-b880-b139113a3d81" (UID: "7ad9afcc-5e14-459a-b880-b139113a3d81"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:16.633423 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.633385    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerID="1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941" exitCode=0
Sep 17 18:31:16.633580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.633434    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82"
Sep 17 18:31:16.633580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.633471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" event={"ID":"7ad9afcc-5e14-459a-b880-b139113a3d81","Type":"ContainerDied","Data":"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"}
Sep 17 18:31:16.633580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.633525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d4f867-8jt82" event={"ID":"7ad9afcc-5e14-459a-b880-b139113a3d81","Type":"ContainerDied","Data":"089e669a9a74ac65ee152119b3c65ca5198fd0d62f89d97c573ae46405bc6245"}
Sep 17 18:31:16.633580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.633546    2559 scope.go:117] "RemoveContainer" containerID="1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"
Sep 17 18:31:16.638051 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638025    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-bound-sa-token\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638051 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638049    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-trusted-ca\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638059    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ad9afcc-5e14-459a-b880-b139113a3d81-ca-trust-extracted\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638069    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-installation-pull-secrets\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638078    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdbl\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-kube-api-access-bfdbl\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638086    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638096    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ad9afcc-5e14-459a-b880-b139113a3d81-registry-certificates\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.638239 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.638108    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ad9afcc-5e14-459a-b880-b139113a3d81-image-registry-private-configuration\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.643048 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.643030    2559 scope.go:117] "RemoveContainer" containerID="1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"
Sep 17 18:31:16.643354 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:31:16.643332    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941\": container with ID starting with 1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941 not found: ID does not exist" containerID="1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"
Sep 17 18:31:16.643423 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.643368    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941"} err="failed to get container status \"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941\": rpc error: code = NotFound desc = could not find container \"1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941\": container with ID starting with 1c39681720c34479b35623c4b7e8e5c6504752582f9b7e3b324c8d7d396f6941 not found: ID does not exist"
Sep 17 18:31:16.668392 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.668363    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:31:16.684111 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:16.684089    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f4d4f867-8jt82"]
Sep 17 18:31:17.945659 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:17.945613    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" path="/var/lib/kubelet/pods/7ad9afcc-5e14-459a-b880-b139113a3d81/volumes"
Sep 17 18:31:19.669436 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:19.669399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:19.755132 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:19.755090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:19.817398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:19.817365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:22.658283 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:22.658246    2559 generic.go:356] "Generic (PLEG): container finished" podID="caef9342-c962-42d4-b237-ed36f6d1a603" containerID="416455199b7d9780eda7f27de4cf53610b600bf16d028edd235a14d0f587f4b3" exitCode=0
Sep 17 18:31:22.658666 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:22.658319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd" event={"ID":"caef9342-c962-42d4-b237-ed36f6d1a603","Type":"ContainerDied","Data":"416455199b7d9780eda7f27de4cf53610b600bf16d028edd235a14d0f587f4b3"}
Sep 17 18:31:22.658760 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:22.658740    2559 scope.go:117] "RemoveContainer" containerID="416455199b7d9780eda7f27de4cf53610b600bf16d028edd235a14d0f587f4b3"
Sep 17 18:31:23.083738 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:23.083696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:23.145362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:23.145329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:23.663930 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:23.663894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gmcpd" event={"ID":"caef9342-c962-42d4-b237-ed36f6d1a603","Type":"ContainerStarted","Data":"14ec8f4e3ca2e6221ee3005398761f626ee3e715d6f9c32311de103f1fc5c5d7"}
Sep 17 18:31:25.719035 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:25.719003    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:31:25.723032 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:25.723010    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-656f8dbc68-lrlwz"
Sep 17 18:31:31.696167 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:31.696134    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce4b99ae-7092-43b0-be9a-74dddc6a6d98" containerID="5811de7cc3244ce424bb9ee9ae0ddc51cb384e2bfc77520683a3f578fdb9a7d1" exitCode=0
Sep 17 18:31:31.696554 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:31.696203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v" event={"ID":"ce4b99ae-7092-43b0-be9a-74dddc6a6d98","Type":"ContainerDied","Data":"5811de7cc3244ce424bb9ee9ae0ddc51cb384e2bfc77520683a3f578fdb9a7d1"}
Sep 17 18:31:31.696554 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:31.696502    2559 scope.go:117] "RemoveContainer" containerID="5811de7cc3244ce424bb9ee9ae0ddc51cb384e2bfc77520683a3f578fdb9a7d1"
Sep 17 18:31:32.702285 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:32.702245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-j5z8v" event={"ID":"ce4b99ae-7092-43b0-be9a-74dddc6a6d98","Type":"ContainerStarted","Data":"1e4c3030a7f2e38aead9928ee22748f95b8e01f2c4f5455c3a958030b58f95d3"}
Sep 17 18:31:47.344762 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:47.344719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:31:47.360765 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:47.360734    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:31:47.773917 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:47.773889    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:31:59.987270 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:31:59.987233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:00.095763 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:00.095710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:05.393790 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.393752    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394091    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="prometheus" containerID="cri-o://e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293" gracePeriod=600
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394104    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy" containerID="cri-o://596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22" gracePeriod=600
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394164    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e" gracePeriod=600
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394173    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="thanos-sidecar" containerID="cri-o://87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f" gracePeriod=600
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394185    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="config-reloader" containerID="cri-o://bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0" gracePeriod=600
Sep 17 18:32:05.394336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.394182    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-web" containerID="cri-o://1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0" gracePeriod=600
Sep 17 18:32:05.632557 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.632530    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:05.745126 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745046    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745126 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745126 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745110    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745167    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shwh4\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745229    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745595 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.745595 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745707 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.745707 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745707 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745708    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745735    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745795    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.745872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745893    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745937    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.745983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746127 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets\") pod \"d31b77f2-1119-427e-a057-07a8cc2d399c\" (UID: \"d31b77f2-1119-427e-a057-07a8cc2d399c\") "
Sep 17 18:32:05.746441 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746282    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.746441 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746300    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-metrics-client-ca\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.746889 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.747045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746959    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.747045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.746972    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.747045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.747003    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:05.747487 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.747321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.748468 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.748398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out" (OuterVolumeSpecName: "config-out") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:05.748577 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.748490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4" (OuterVolumeSpecName: "kube-api-access-shwh4") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "kube-api-access-shwh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:05.748577 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.748549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.749094 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.748761    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config" (OuterVolumeSpecName: "config") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.749315 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.749287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.749477 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.749446    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.749715 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.749688    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.749715 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.749702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.749910 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.749779    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.750155 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.750135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.750331 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.750318    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:05.751292 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.751276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.761184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.761160    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config" (OuterVolumeSpecName: "web-config") pod "d31b77f2-1119-427e-a057-07a8cc2d399c" (UID: "d31b77f2-1119-427e-a057-07a8cc2d399c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.833537 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833498    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e" exitCode=0
Sep 17 18:32:05.833537 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833524    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22" exitCode=0
Sep 17 18:32:05.833537 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833533    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0" exitCode=0
Sep 17 18:32:05.833537 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833542    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f" exitCode=0
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833549    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0" exitCode=0
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833557    2559 generic.go:356] "Generic (PLEG): container finished" podID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293" exitCode=0
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833725    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"}
Sep 17 18:32:05.833894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.833781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d31b77f2-1119-427e-a057-07a8cc2d399c","Type":"ContainerDied","Data":"71aaaea9827a293c77724c0ea7634411cdc9b63ea8b56e617b5946243c68b79b"}
Sep 17 18:32:05.841903 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.841669    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.847818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847796    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847817    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-grpc-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847842    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-kube-rbac-proxy\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847853    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847862    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847871    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-metrics-client-certs\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847897    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-web-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847906    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847915    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847923    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847931    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847940    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-tls-assets\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847948    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847955    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d31b77f2-1119-427e-a057-07a8cc2d399c-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847963    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwh4\" (UniqueName: \"kubernetes.io/projected/d31b77f2-1119-427e-a057-07a8cc2d399c-kube-api-access-shwh4\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847970    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-prometheus-k8s-db\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.847999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.847977    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d31b77f2-1119-427e-a057-07a8cc2d399c-config-out\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.848739 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.848716    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.855392 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.855375    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.858162 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.858140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:05.863423 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.863403    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.863970 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.863947    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:05.870281 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.870265    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.877313 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.877297    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.884122 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884106    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.884374 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.884355    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.884434 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.884434 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884402    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.884629 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.884612    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.884672 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884633    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.884672 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884645    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.884908 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.884888    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.884970 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884911    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.884970 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.884928    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.885144 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.885128    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.885184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885146    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.885184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885157    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.885356 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.885341    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.885398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885360    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.885398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885372    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.885577 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.885559    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.885622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885588    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.885622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885600    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.885788 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:32:05.885772    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.885848 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.885848 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.885804    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.886105 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886083    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.886167 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886106    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.886358 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886342    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.886398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886358    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.886598 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886581    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.886647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886598    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.886858 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886816    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.886937 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.886861    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.887121 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887106    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.887172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887123    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.887329 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887311    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.887397 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887331    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.887545 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887530    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.887596 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887546    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.887758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887741    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.887824 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887759    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.887995 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.888049 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.887996    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.888179 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888165    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.888231 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888180    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.888376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888361    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.888426 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888377    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.888551 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888536    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.888602 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888551    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.888728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888712    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.888770 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888728    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.888960 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888944    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.888960 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.888959    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.889154 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889137    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.889209 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889156    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.889417 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889397    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.889504 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889418    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.889665 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889644    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.889739 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889667    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.889915 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889898    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.889969 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.889915    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.890119 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890097    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.890186 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890119    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.890338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890322    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.890393 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890338    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.890593 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890546    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.890593 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890590    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.890981 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890879    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.891059 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.890983    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.891535 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.891509    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.891616 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.891537    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.891796 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.891778    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.891879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.891798    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.892049 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892027    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.892049 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892045    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.892308 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892282    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.892368 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892310    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.892539 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892513    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.892602 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892539    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.892791 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892771    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.892896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892793    2559 scope.go:117] "RemoveContainer" containerID="348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"
Sep 17 18:32:05.892896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.892849    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:05.893150 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893127    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e"} err="failed to get container status \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": rpc error: code = NotFound desc = could not find container \"348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e\": container with ID starting with 348bc530b7d230944894f4e365d0a7b70a298f72b92ced98318eb9f3aa62f89e not found: ID does not exist"
Sep 17 18:32:05.893194 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893151    2559 scope.go:117] "RemoveContainer" containerID="596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"
Sep 17 18:32:05.893194 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893164    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="prometheus"
Sep 17 18:32:05.893194 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893177    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="prometheus"
Sep 17 18:32:05.893194 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893192    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-web"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893198    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-web"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893204    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="thanos-sidecar"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893210    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="thanos-sidecar"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893228    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-thanos"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893233    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-thanos"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893242    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerName="registry"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893249    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerName="registry"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893259    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893266    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893278    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="init-config-reloader"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893285    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="init-config-reloader"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893292    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="config-reloader"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893300    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="config-reloader"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893353    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad9afcc-5e14-459a-b880-b139113a3d81" containerName="registry"
Sep 17 18:32:05.893362 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-thanos"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893373    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="config-reloader"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893383    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy-web"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893388    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="kube-rbac-proxy"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893401    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="prometheus"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893406    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" containerName="thanos-sidecar"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22"} err="failed to get container status \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": rpc error: code = NotFound desc = could not find container \"596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22\": container with ID starting with 596b76ebc72f2181215c00da4dc99a33c8f88ffb120e9210fb6ddd3b90540d22 not found: ID does not exist"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893471    2559 scope.go:117] "RemoveContainer" containerID="1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893713    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0"} err="failed to get container status \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": rpc error: code = NotFound desc = could not find container \"1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0\": container with ID starting with 1011e6cacd635ab1743ef3e37895b2f72cabc475f890b27648de2ebcfa6f75c0 not found: ID does not exist"
Sep 17 18:32:05.893879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893725    2559 scope.go:117] "RemoveContainer" containerID="87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"
Sep 17 18:32:05.894220 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.893991    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f"} err="failed to get container status \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": rpc error: code = NotFound desc = could not find container \"87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f\": container with ID starting with 87da4792177606facb64d6115e1f27cae0b84725066b11841850c819d27ef53f not found: ID does not exist"
Sep 17 18:32:05.894220 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894016    2559 scope.go:117] "RemoveContainer" containerID="bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"
Sep 17 18:32:05.894304 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894240    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0"} err="failed to get container status \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": rpc error: code = NotFound desc = could not find container \"bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0\": container with ID starting with bdb8b4d9c1da430ee532ae692a581d4baddb02814c8a8af8e1155f28a78616b0 not found: ID does not exist"
Sep 17 18:32:05.894304 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894254    2559 scope.go:117] "RemoveContainer" containerID="e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"
Sep 17 18:32:05.894482 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894463    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293"} err="failed to get container status \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": rpc error: code = NotFound desc = could not find container \"e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293\": container with ID starting with e8c4170d1f451753f37f4ec40d6f2ef5a9375f10a539eb71a809fa3a9a1ca293 not found: ID does not exist"
Sep 17 18:32:05.894557 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894483    2559 scope.go:117] "RemoveContainer" containerID="aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"
Sep 17 18:32:05.894713 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.894696    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202"} err="failed to get container status \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": rpc error: code = NotFound desc = could not find container \"aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202\": container with ID starting with aa9e75339fc9f978dcfeb945c2c38aef6ce54d95f39850a7cf8f9e5c9f510202 not found: ID does not exist"
Sep 17 18:32:05.898397 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.898382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:05.900997 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.900978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:32:05.902113 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:32:05.902247 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902229    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:32:05.902571 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902555    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:32:05.902657 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902615    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:32:05.902710 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:32:05.902851 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902821    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5hsd27l5ep16f\""
Sep 17 18:32:05.902935 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.902874    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:32:05.903050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903029    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:32:05.903125 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:32:05.903471 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903450    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:32:05.903562 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903509    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:32:05.903562 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903554    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vpw9j\""
Sep 17 18:32:05.903754 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.903733    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:32:05.906180 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.906163    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:32:05.909652 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.909637    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:32:05.915124 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.915104    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:05.940757 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:05.940725    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31b77f2-1119-427e-a057-07a8cc2d399c" path="/var/lib/kubelet/pods/d31b77f2-1119-427e-a057-07a8cc2d399c/volumes"
Sep 17 18:32:06.055220 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xm6\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-kube-api-access-v4xm6\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055403 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055403 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055403 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055403 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055403 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055556 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.055740 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.055732    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.156851 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.156851 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.156978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157091 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xm6\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-kube-api-access-v4xm6\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.157638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.158028 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.157949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.158559 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.158247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160183 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160183 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160344 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160344 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160344 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.160918 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161019 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.160916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.161120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161217 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.161170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161380 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.161356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161648 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.161624    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.161728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.161685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.162315 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.162286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.162399 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.162383    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.162923 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.162905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-config\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.163014 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.162998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.172286 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.172262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xm6\" (UniqueName: \"kubernetes.io/projected/34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20-kube-api-access-v4xm6\") pod \"prometheus-k8s-0\" (UID: \"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.208267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.208234    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:06.492531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.492508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:32:06.494065 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:32:06.494039    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b7af9d_1e61_42c8_9b1c_6f5fe8b33b20.slice/crio-1e1a6b189d5bbd54fb2540c6af5acc58c7f1a26c70ae0e9c87d43c97eca3af0e WatchSource:0}: Error finding container 1e1a6b189d5bbd54fb2540c6af5acc58c7f1a26c70ae0e9c87d43c97eca3af0e: Status 404 returned error can't find the container with id 1e1a6b189d5bbd54fb2540c6af5acc58c7f1a26c70ae0e9c87d43c97eca3af0e
Sep 17 18:32:06.839963 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.839925    2559 generic.go:356] "Generic (PLEG): container finished" podID="34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20" containerID="bf683585163f665f307762e1f0f96be5c11cc30b5e685315d391b1eb1f604274" exitCode=0
Sep 17 18:32:06.840141 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.840006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerDied","Data":"bf683585163f665f307762e1f0f96be5c11cc30b5e685315d391b1eb1f604274"}
Sep 17 18:32:06.840141 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.840075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"1e1a6b189d5bbd54fb2540c6af5acc58c7f1a26c70ae0e9c87d43c97eca3af0e"}
Sep 17 18:32:06.948419 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:06.948393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:07.703741 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.703655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:07.850398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"443d375e4cd4f6509ffda09b2efd6f26da611712b1604c15c159888e5594fc80"}
Sep 17 18:32:07.850563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"466a13aff8a75d25df165698209d6cbe3b8e2ac7d30f5546e0a3ee9d5bc69373"}
Sep 17 18:32:07.850563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"25a16a3f34e3d4433696ca203c12c99968fa07ec70015a23ef0e07796089cce8"}
Sep 17 18:32:07.850563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"32e2d8044b028c122a799e5cb29d178a37cd8c854db774e2376595377c2c7488"}
Sep 17 18:32:07.850563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"34b669f3f468a962e17025cda9852db53a1316f2f540d1319efc53a16e94b5cb"}
Sep 17 18:32:07.850563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.850527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20","Type":"ContainerStarted","Data":"16b19f8bc09904027b1ea98abe129f20ae3b135dbc18b2aa4c0c6c59f4a1ab35"}
Sep 17 18:32:07.893132 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:07.893078    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.87470304 podStartE2EDuration="2.893059603s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.356563ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:07.890360851 +0000 UTC m=+170.440584737" watchObservedRunningTime="2026-09-17 18:32:07.893059603 +0000 UTC m=+170.443283500"
Sep 17 18:32:08.808236 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:08.808186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:08.908237 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:08.908205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:11.208597 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:11.208515    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:32:19.852454 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:19.852416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:19.907259 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:19.907224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:20.050030 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:20.049999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:53.715458 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:53.715421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:53.774548 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:53.774517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:56.573535 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:32:56.573497    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:06.208675 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:06.208628    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:33:06.223382 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:06.223352    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:33:07.083846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:07.083799    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:33:15.266518 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:15.266483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:15.367695 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:15.367661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:16.155169 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:16.155104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:16.210322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:16.210291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:27.127306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:27.127269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:27.228630 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:27.228584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:27.295432 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:27.295403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:34.441108 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:34.441072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:38.113586 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:38.113553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:38.217420 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:33:38.217382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:17.815674 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:17.815642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:34:17.818143 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:17.818120    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:34:17.826081 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:17.826062    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:30.286797 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:30.286754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:32.409678 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:32.409645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:49.935784 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:49.935744    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:50.010440 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:50.010393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:58.164563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:58.164527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:58.253003 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:34:58.252969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:03.516876 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:03.516840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:03.808441 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:03.808353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.006941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.006903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.110959 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.110879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.563483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.563442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.681006 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.680974    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.770685 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.770655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.872021 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.871911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.979532 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:04.979496    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.072254 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:05.072222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.282950 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:05.282919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.381870 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:05.381813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.482313 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:05.482281    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:36.422681 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:36.422594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:36.492438 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:36.492407    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:47.833570 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:47.833528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:47.933456 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:47.933420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:47.993052 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:47.993004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:56.292097 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:56.292013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:56.345309 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:35:56.345271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.182172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:24.182131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.301457 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:24.301427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.434175 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:24.434099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.561880 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:24.561849    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:25.498140 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:25.498098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:25.724499 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:36:25.724466    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.425715 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.425680    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.466734 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.466705    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-kkjbb"]
Sep 17 18:37:44.469910 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.469894    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.474474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.474451    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:37:44.474654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.474641    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:37:44.474734 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.474716    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-td88r\""
Sep 17 18:37:44.475446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.475430    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:37:44.490140 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.490111    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-kkjbb"]
Sep 17 18:37:44.515078 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.515043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.594084 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.594043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strrv\" (UniqueName: \"kubernetes.io/projected/75c467cf-84ff-485e-80f1-77d77d327d18-kube-api-access-strrv\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.594272 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.594095    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.598056 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.598022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.695408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.695327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strrv\" (UniqueName: \"kubernetes.io/projected/75c467cf-84ff-485e-80f1-77d77d327d18-kube-api-access-strrv\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.695408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.695385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.695625 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:37:44.695521    2559 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:37:44.695625 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:37:44.695588    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs podName:75c467cf-84ff-485e-80f1-77d77d327d18 nodeName:}" failed. No retries permitted until 2026-09-17 18:37:45.195565969 +0000 UTC m=+507.745789835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs") pod "model-serving-api-7b954ffc95-kkjbb" (UID: "75c467cf-84ff-485e-80f1-77d77d327d18") : secret "model-serving-api-tls" not found
Sep 17 18:37:44.696166 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.696146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.706699 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.706675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-strrv\" (UniqueName: \"kubernetes.io/projected/75c467cf-84ff-485e-80f1-77d77d327d18-kube-api-access-strrv\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:44.796481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:44.796450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:45.200015 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.199960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:45.202386 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.202356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c467cf-84ff-485e-80f1-77d77d327d18-tls-certs\") pod \"model-serving-api-7b954ffc95-kkjbb\" (UID: \"75c467cf-84ff-485e-80f1-77d77d327d18\") " pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:45.380022 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.379989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:45.587678 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.587642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:45.650710 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.650677    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-kkjbb"]
Sep 17 18:37:45.653711 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:37:45.653680    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c467cf_84ff_485e_80f1_77d77d327d18.slice/crio-2ff5a082a73c7487588a979bf284803d6622cf3745b18872351ed2d634cd975d WatchSource:0}: Error finding container 2ff5a082a73c7487588a979bf284803d6622cf3745b18872351ed2d634cd975d: Status 404 returned error can't find the container with id 2ff5a082a73c7487588a979bf284803d6622cf3745b18872351ed2d634cd975d
Sep 17 18:37:45.656074 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.656058    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:37:45.667405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:45.667375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:46.127714 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:46.127681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-kkjbb" event={"ID":"75c467cf-84ff-485e-80f1-77d77d327d18","Type":"ContainerStarted","Data":"2ff5a082a73c7487588a979bf284803d6622cf3745b18872351ed2d634cd975d"}
Sep 17 18:37:48.137679 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:48.137581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-kkjbb" event={"ID":"75c467cf-84ff-485e-80f1-77d77d327d18","Type":"ContainerStarted","Data":"8519ddb6ace6969e42ae42bfc6bfcf7df48812493887755dadd50b6b5bdbb24c"}
Sep 17 18:37:48.138083 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:48.137692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:37:48.157420 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:48.157370    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-kkjbb" podStartSLOduration=2.014093031 podStartE2EDuration="4.157354841s" totalImagesPullingTime="2.14326181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:48.155368412 +0000 UTC m=+510.705592297" watchObservedRunningTime="2026-09-17 18:37:48.157354841 +0000 UTC m=+510.707578727"
Sep 17 18:37:56.940070 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:56.940033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:59.146092 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:37:59.146049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-kkjbb"
Sep 17 18:39:17.843641 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:17.843609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:39:17.844533 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:17.844510    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:39:20.536550 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:20.536512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:20.646083 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:20.646048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.852999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:22.852966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:32.654991 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:32.654952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:33.599098 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.599066    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:39:33.601684 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.601663    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.604872 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.604850    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2hvk\""
Sep 17 18:39:33.605006 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.604992    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:33.605787 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.605768    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:33.605941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.605841    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e0a45-kube-rbac-proxy-sar-config\""
Sep 17 18:39:33.605941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.605855    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e0a45-predictor-serving-cert\""
Sep 17 18:39:33.623335 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.623295    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:39:33.699682 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.699644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n2tm\" (UniqueName: \"kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.700079 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.699720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.700079 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.699748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.800572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.800541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.800572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.800580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.800787 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.800638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n2tm\" (UniqueName: \"kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.801219 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.801198    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.803118 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.803101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.847902 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.847875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n2tm\" (UniqueName: \"kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm\") pod \"error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") " pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:33.911956 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:33.911867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:34.123702 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:34.123665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:34.211268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:34.211178    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:39:34.213722 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:39:34.213690    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661030a1_570e_4b0b_8d9e_8fec338a1aa5.slice/crio-5814090295d5eea17d96dd29400b19f0cc4335acda8ca9e1aec1660da1345bcf WatchSource:0}: Error finding container 5814090295d5eea17d96dd29400b19f0cc4335acda8ca9e1aec1660da1345bcf: Status 404 returned error can't find the container with id 5814090295d5eea17d96dd29400b19f0cc4335acda8ca9e1aec1660da1345bcf
Sep 17 18:39:34.457750 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:34.457718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:34.549856 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:34.549803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerStarted","Data":"5814090295d5eea17d96dd29400b19f0cc4335acda8ca9e1aec1660da1345bcf"}
Sep 17 18:39:46.614567 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:46.614521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerStarted","Data":"c762b2ec87a9447b19e86ecbdd32485361cd5ed6b7bfb60e007e884d207b22c3"}
Sep 17 18:39:48.625121 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:48.625084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerStarted","Data":"82f6a8c52f418acbfeafe0a281898aa86121204c387c440510d31f7e04f502dc"}
Sep 17 18:39:48.625531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:48.625193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:48.643149 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:48.643092    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podStartSLOduration=1.486691249 podStartE2EDuration="15.643075804s" totalImagesPullingTime="14.156384555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:48.64154433 +0000 UTC m=+631.191768231" watchObservedRunningTime="2026-09-17 18:39:48.643075804 +0000 UTC m=+631.193299691"
Sep 17 18:39:49.629216 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:49.629184    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:49.630231 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:49.630199    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:39:50.633429 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:50.633389    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:39:55.638178 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:55.638146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:39:55.638678 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:39:55.638651    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:40:05.639582 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:05.639541    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:40:13.618497 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:13.618464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:13.733229 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:13.733197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.639532 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:15.639488    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:40:25.638922 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:25.638882    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:40:35.639900 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:40:35.639868    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:41:12.626568 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.626492    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:12.629761 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.629745    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.631295 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.631268    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:12.632233 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.632216    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e0a45-serving-cert\""
Sep 17 18:41:12.632297 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.632233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e0a45-kube-rbac-proxy-sar-config\""
Sep 17 18:41:12.637908 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.637884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:12.754631 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.754596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.754805 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.754683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.856141 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.856100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.856315 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.856183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.856315 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:41:12.856305    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-e0a45-serving-cert: secret "switch-graph-e0a45-serving-cert" not found
Sep 17 18:41:12.856395 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:41:12.856388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls podName:e4bf1d82-ce3b-46b1-ad39-7f217edd6641 nodeName:}" failed. No retries permitted until 2026-09-17 18:41:13.356363468 +0000 UTC m=+715.906587347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls") pod "switch-graph-e0a45-9684bdf46-n62b8" (UID: "e4bf1d82-ce3b-46b1-ad39-7f217edd6641") : secret "switch-graph-e0a45-serving-cert" not found
Sep 17 18:41:12.856850 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.856811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:12.910089 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:12.910012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:13.359063 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:13.359022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:13.361419 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:13.361392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") pod \"switch-graph-e0a45-9684bdf46-n62b8\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") " pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:13.540976 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:13.540942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:13.808531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:13.808499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:13.811397 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:41:13.811366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bf1d82_ce3b_46b1_ad39_7f217edd6641.slice/crio-981ddb6ae9b0e0569a50788ccef771782f3a94e242ba9afdd16e4c54abd4af6d WatchSource:0}: Error finding container 981ddb6ae9b0e0569a50788ccef771782f3a94e242ba9afdd16e4c54abd4af6d: Status 404 returned error can't find the container with id 981ddb6ae9b0e0569a50788ccef771782f3a94e242ba9afdd16e4c54abd4af6d
Sep 17 18:41:13.962586 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:13.962551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" event={"ID":"e4bf1d82-ce3b-46b1-ad39-7f217edd6641","Type":"ContainerStarted","Data":"981ddb6ae9b0e0569a50788ccef771782f3a94e242ba9afdd16e4c54abd4af6d"}
Sep 17 18:41:14.618304 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:14.618231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:14.675232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:14.675134    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:16.977879 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:16.977811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" event={"ID":"e4bf1d82-ce3b-46b1-ad39-7f217edd6641","Type":"ContainerStarted","Data":"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"}
Sep 17 18:41:16.978363 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:16.977994    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:16.994524 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:16.994479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podStartSLOduration=2.561961486 podStartE2EDuration="4.994463812s" totalImagesPullingTime="2.432502326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:16.993976838 +0000 UTC m=+719.544200727" watchObservedRunningTime="2026-09-17 18:41:16.994463812 +0000 UTC m=+719.544687698"
Sep 17 18:41:22.989728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:22.989695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:26.748366 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.748327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:26.748735 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.748581    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" containerID="cri-o://36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2" gracePeriod=30
Sep 17 18:41:26.798876 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.798825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:26.960065 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.960031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:26.967459 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.967431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:41:26.967693 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.967671    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container" containerID="cri-o://c762b2ec87a9447b19e86ecbdd32485361cd5ed6b7bfb60e007e884d207b22c3" gracePeriod=30
Sep 17 18:41:26.967771 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:26.967699    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kube-rbac-proxy" containerID="cri-o://82f6a8c52f418acbfeafe0a281898aa86121204c387c440510d31f7e04f502dc" gracePeriod=30
Sep 17 18:41:27.060195 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.060102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.161677 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.161640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.169118 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.169090    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:41:27.172371 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.172355    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.174707 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.174681    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5f0c-predictor-serving-cert\""
Sep 17 18:41:27.174813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.174741    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5f0c-kube-rbac-proxy-sar-config\""
Sep 17 18:41:27.180388 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.180360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.180514 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.180409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.180514 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.180447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87ct\" (UniqueName: \"kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.182559 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.182537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:41:27.261197 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.261159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.281358 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.281330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.281481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.281369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.281481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.281388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87ct\" (UniqueName: \"kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.281582 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:41:27.281490    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-serving-cert: secret "error-404-isvc-c5f0c-predictor-serving-cert" not found
Sep 17 18:41:27.281582 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:41:27.281550    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls podName:dcd5d47d-4201-4f9d-ba5b-2220c662ebeb nodeName:}" failed. No retries permitted until 2026-09-17 18:41:27.781531433 +0000 UTC m=+730.331755315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls") pod "error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" (UID: "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb") : secret "error-404-isvc-c5f0c-predictor-serving-cert" not found
Sep 17 18:41:27.282007 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.281990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.290505 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.290471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87ct\" (UniqueName: \"kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.360285 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.360191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.540211 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.540169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.785192 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.785160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.787596 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.787578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") pod \"error-404-isvc-c5f0c-predictor-78b5f64bc-74flm\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") " pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:27.988310 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:27.988273    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:28.024260 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:28.024215    2559 generic.go:356] "Generic (PLEG): container finished" podID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerID="82f6a8c52f418acbfeafe0a281898aa86121204c387c440510d31f7e04f502dc" exitCode=2
Sep 17 18:41:28.024626 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:28.024281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerDied","Data":"82f6a8c52f418acbfeafe0a281898aa86121204c387c440510d31f7e04f502dc"}
Sep 17 18:41:28.083490 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:28.083406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:28.354468 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:28.354388    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:41:28.357432 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:41:28.357402    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd5d47d_4201_4f9d_ba5b_2220c662ebeb.slice/crio-549f392d446f1824ba9079edbf7ca51b2bc8f0697878b45a594a8da92f93d6c1 WatchSource:0}: Error finding container 549f392d446f1824ba9079edbf7ca51b2bc8f0697878b45a594a8da92f93d6c1: Status 404 returned error can't find the container with id 549f392d446f1824ba9079edbf7ca51b2bc8f0697878b45a594a8da92f93d6c1
Sep 17 18:41:29.030377 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:29.030344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerStarted","Data":"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"}
Sep 17 18:41:29.030778 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:29.030398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerStarted","Data":"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"}
Sep 17 18:41:29.030778 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:29.030427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerStarted","Data":"549f392d446f1824ba9079edbf7ca51b2bc8f0697878b45a594a8da92f93d6c1"}
Sep 17 18:41:29.030778 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:29.030499    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:29.048863 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:29.048798    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podStartSLOduration=2.048781195 podStartE2EDuration="2.048781195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:29.048194366 +0000 UTC m=+731.598418253" watchObservedRunningTime="2026-09-17 18:41:29.048781195 +0000 UTC m=+731.599005081"
Sep 17 18:41:30.036807 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.036779    2559 generic.go:356] "Generic (PLEG): container finished" podID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerID="c762b2ec87a9447b19e86ecbdd32485361cd5ed6b7bfb60e007e884d207b22c3" exitCode=0
Sep 17 18:41:30.037294 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.036857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerDied","Data":"c762b2ec87a9447b19e86ecbdd32485361cd5ed6b7bfb60e007e884d207b22c3"}
Sep 17 18:41:30.037294 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.037163    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:30.038271 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.038244    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:41:30.110534 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.110511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:41:30.204898 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.204787    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls\") pod \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") "
Sep 17 18:41:30.204898 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.204892    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config\") pod \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") "
Sep 17 18:41:30.205080 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.204925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n2tm\" (UniqueName: \"kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm\") pod \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\" (UID: \"661030a1-570e-4b0b-8d9e-8fec338a1aa5\") "
Sep 17 18:41:30.205240 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.205213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "661030a1-570e-4b0b-8d9e-8fec338a1aa5" (UID: "661030a1-570e-4b0b-8d9e-8fec338a1aa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:30.207093 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.207062    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "661030a1-570e-4b0b-8d9e-8fec338a1aa5" (UID: "661030a1-570e-4b0b-8d9e-8fec338a1aa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:30.207195 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.207124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm" (OuterVolumeSpecName: "kube-api-access-4n2tm") pod "661030a1-570e-4b0b-8d9e-8fec338a1aa5" (UID: "661030a1-570e-4b0b-8d9e-8fec338a1aa5"). InnerVolumeSpecName "kube-api-access-4n2tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:30.306458 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.306423    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.306458 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.306450    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n2tm\" (UniqueName: \"kubernetes.io/projected/661030a1-570e-4b0b-8d9e-8fec338a1aa5-kube-api-access-4n2tm\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.306458 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:30.306459    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661030a1-570e-4b0b-8d9e-8fec338a1aa5-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:41:31.043288 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.043256    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"
Sep 17 18:41:31.043741 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.043285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh" event={"ID":"661030a1-570e-4b0b-8d9e-8fec338a1aa5","Type":"ContainerDied","Data":"5814090295d5eea17d96dd29400b19f0cc4335acda8ca9e1aec1660da1345bcf"}
Sep 17 18:41:31.043741 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.043349    2559 scope.go:117] "RemoveContainer" containerID="82f6a8c52f418acbfeafe0a281898aa86121204c387c440510d31f7e04f502dc"
Sep 17 18:41:31.043907 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.043881    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:41:31.052618 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.052588    2559 scope.go:117] "RemoveContainer" containerID="c762b2ec87a9447b19e86ecbdd32485361cd5ed6b7bfb60e007e884d207b22c3"
Sep 17 18:41:31.066776 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.066750    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:41:31.073259 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.073234    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0a45-predictor-8545c9dc7f-pqqjh"]
Sep 17 18:41:31.940160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:31.940128    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" path="/var/lib/kubelet/pods/661030a1-570e-4b0b-8d9e-8fec338a1aa5/volumes"
Sep 17 18:41:32.987416 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:32.987374    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:36.048015 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:36.047984    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:41:36.048573 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:36.048547    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:41:37.988630 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:37.988585    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:37.989111 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:37.988694    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:42.987915 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:42.987866    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:46.048804 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:46.048756    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:41:47.988232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:47.988199    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:52.594758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.594718    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:41:52.595223 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595179    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kube-rbac-proxy"
Sep 17 18:41:52.595223 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595199    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kube-rbac-proxy"
Sep 17 18:41:52.595339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595237    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container"
Sep 17 18:41:52.595339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595246    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container"
Sep 17 18:41:52.595339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595326    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kserve-container"
Sep 17 18:41:52.595339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.595338    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="661030a1-570e-4b0b-8d9e-8fec338a1aa5" containerName="kube-rbac-proxy"
Sep 17 18:41:52.599866 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.599817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.600128 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.600093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:52.602287 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.602267    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 18:41:52.602405 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.602377    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 18:41:52.608117 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.608093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:41:52.692249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.692204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.692480 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.692372    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.723738 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.723691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:52.793471 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.793437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.793630 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.793480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.794101 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.794074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.795900 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.795870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls\") pod \"model-chainer-6f687ffc4-z65q8\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") " pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.910145 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.910064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:52.987779 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:52.987745    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:53.195680 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:53.195609    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:41:53.198965 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:41:53.198935    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a04d9a2_1eea_41ba_8a29_73a61d75a891.slice/crio-fe5c915134400f2bce94aae69ad1fe5efced91dbc974ef86a57af0b74e7b293f WatchSource:0}: Error finding container fe5c915134400f2bce94aae69ad1fe5efced91dbc974ef86a57af0b74e7b293f: Status 404 returned error can't find the container with id fe5c915134400f2bce94aae69ad1fe5efced91dbc974ef86a57af0b74e7b293f
Sep 17 18:41:54.136360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:54.136324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" event={"ID":"6a04d9a2-1eea-41ba-8a29-73a61d75a891","Type":"ContainerStarted","Data":"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"}
Sep 17 18:41:54.136812 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:54.136371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" event={"ID":"6a04d9a2-1eea-41ba-8a29-73a61d75a891","Type":"ContainerStarted","Data":"fe5c915134400f2bce94aae69ad1fe5efced91dbc974ef86a57af0b74e7b293f"}
Sep 17 18:41:54.136812 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:54.136443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:41:54.157042 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:54.156992    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podStartSLOduration=2.156977641 podStartE2EDuration="2.156977641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:54.155414261 +0000 UTC m=+756.705638152" watchObservedRunningTime="2026-09-17 18:41:54.156977641 +0000 UTC m=+756.707201531"
Sep 17 18:41:56.048648 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:56.048600    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:41:56.917612 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:56.917587    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:57.028444 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.028353    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle\") pod \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") "
Sep 17 18:41:57.028444 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.028435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") pod \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\" (UID: \"e4bf1d82-ce3b-46b1-ad39-7f217edd6641\") "
Sep 17 18:41:57.028818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.028783    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e4bf1d82-ce3b-46b1-ad39-7f217edd6641" (UID: "e4bf1d82-ce3b-46b1-ad39-7f217edd6641"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:57.030541 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.030513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4bf1d82-ce3b-46b1-ad39-7f217edd6641" (UID: "e4bf1d82-ce3b-46b1-ad39-7f217edd6641"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:57.129026 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.128991    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:41:57.129026 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.129024    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4bf1d82-ce3b-46b1-ad39-7f217edd6641-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:41:57.149564 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.149528    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerID="36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2" exitCode=0
Sep 17 18:41:57.149730 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.149615    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"
Sep 17 18:41:57.149730 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.149617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" event={"ID":"e4bf1d82-ce3b-46b1-ad39-7f217edd6641","Type":"ContainerDied","Data":"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"}
Sep 17 18:41:57.149815 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.149742    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8" event={"ID":"e4bf1d82-ce3b-46b1-ad39-7f217edd6641","Type":"ContainerDied","Data":"981ddb6ae9b0e0569a50788ccef771782f3a94e242ba9afdd16e4c54abd4af6d"}
Sep 17 18:41:57.149815 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.149764    2559 scope.go:117] "RemoveContainer" containerID="36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"
Sep 17 18:41:57.158402 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.158383    2559 scope.go:117] "RemoveContainer" containerID="36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"
Sep 17 18:41:57.158694 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:41:57.158673    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2\": container with ID starting with 36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2 not found: ID does not exist" containerID="36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"
Sep 17 18:41:57.158768 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.158706    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2"} err="failed to get container status \"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2\": rpc error: code = NotFound desc = could not find container \"36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2\": container with ID starting with 36ded6db798c7159c8ee25f50bb4d553b46fcebcac8bd09424ccdaca460aede2 not found: ID does not exist"
Sep 17 18:41:57.171354 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.171330    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:57.175028 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.175007    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e0a45-9684bdf46-n62b8"]
Sep 17 18:41:57.940646 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:41:57.940615    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" path="/var/lib/kubelet/pods/e4bf1d82-ce3b-46b1-ad39-7f217edd6641/volumes"
Sep 17 18:42:00.146401 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:00.146373    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:42:03.147080 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.147043    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:42:03.147555 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.147356    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" containerID="cri-o://922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7" gracePeriod=30
Sep 17 18:42:03.181336 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.181290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.278504 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.278278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.354927 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.354889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.441368 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.441289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.518226 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.518180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.527482 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.527450    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:42:03.527767 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.527755    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45"
Sep 17 18:42:03.527808 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.527769    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45"
Sep 17 18:42:03.527860 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.527838    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bf1d82-ce3b-46b1-ad39-7f217edd6641" containerName="switch-graph-e0a45"
Sep 17 18:42:03.532092 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.532075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.535623 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.535584    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-65928-kube-rbac-proxy-sar-config\""
Sep 17 18:42:03.535730 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.535639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-65928-predictor-serving-cert\""
Sep 17 18:42:03.541228 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.541207    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:42:03.618199 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.618165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.680082 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.680043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.680291 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.680100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.680291 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.680217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psqm\" (UniqueName: \"kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.739487 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.739399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:03.781442 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.781404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psqm\" (UniqueName: \"kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.781639 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.781459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.781639 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.781492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.782207 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.782184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.783963 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.783943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.790232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.790207    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psqm\" (UniqueName: \"kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm\") pod \"error-404-isvc-65928-predictor-6b77457dbf-g7kr9\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") " pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:03.842990 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:03.842959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:04.117724 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:04.117694    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:42:04.120645 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:42:04.120616    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ad6913_e42a_4ee3_8e5e_804a529bd5ea.slice/crio-1682080326ed851a6eb3e78ac79962f7a766de224958e6ebe4218127cefc4960 WatchSource:0}: Error finding container 1682080326ed851a6eb3e78ac79962f7a766de224958e6ebe4218127cefc4960: Status 404 returned error can't find the container with id 1682080326ed851a6eb3e78ac79962f7a766de224958e6ebe4218127cefc4960
Sep 17 18:42:04.180199 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:04.180169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerStarted","Data":"1682080326ed851a6eb3e78ac79962f7a766de224958e6ebe4218127cefc4960"}
Sep 17 18:42:05.145338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:05.145302    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:05.186664 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:05.186627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerStarted","Data":"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"}
Sep 17 18:42:05.187118 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:05.186696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerStarted","Data":"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"}
Sep 17 18:42:05.187118 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:05.186731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:05.205963 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:05.205910    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podStartSLOduration=2.20589361 podStartE2EDuration="2.20589361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:05.203477973 +0000 UTC m=+767.753701860" watchObservedRunningTime="2026-09-17 18:42:05.20589361 +0000 UTC m=+767.756117497"
Sep 17 18:42:06.049474 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:06.049428    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 18:42:06.191059 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:06.191025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:06.192495 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:06.192458    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:07.195298 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:07.195262    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:08.898446 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:08.898411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:08.991243 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:08.991199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:10.144742 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:10.144707    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:12.199764 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:12.199735    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:42:12.200262 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:12.200235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:15.144998 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:15.144959    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:15.145408 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:15.145076    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:42:16.049240 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:16.049199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:42:20.144430 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:20.144386    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:22.200171 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:22.200134    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:25.145295 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:25.145248    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:27.034975 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.034935    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:42:27.039268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.039250    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.041186 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.041164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:27.041606 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.041588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c5f0c-serving-cert\""
Sep 17 18:42:27.041694 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.041590    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c5f0c-kube-rbac-proxy-sar-config\""
Sep 17 18:42:27.047367 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.047342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:42:27.073999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.073968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.073999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.073999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.175172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.175137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.175357 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.175231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.175357 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:42:27.175307    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-c5f0c-serving-cert: secret "switch-graph-c5f0c-serving-cert" not found
Sep 17 18:42:27.175446 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:42:27.175373    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls podName:8c7b6c22-dea2-47ce-8e1b-7fce54053f92 nodeName:}" failed. No retries permitted until 2026-09-17 18:42:27.675353403 +0000 UTC m=+790.225577270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls") pod "switch-graph-c5f0c-844f475959-vnjtz" (UID: "8c7b6c22-dea2-47ce-8e1b-7fce54053f92") : secret "switch-graph-c5f0c-serving-cert" not found
Sep 17 18:42:27.175823 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.175804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.280672 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.280643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:27.679314 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.679280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.681700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.681667    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") pod \"switch-graph-c5f0c-844f475959-vnjtz\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") " pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:27.950360 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:27.950272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:28.223156 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:28.223131    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:42:28.225454 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:42:28.225413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7b6c22_dea2_47ce_8e1b_7fce54053f92.slice/crio-26ea8bbfe8e9b6b619150c33a6fbb0e586f91bcb4397f12d0c1d09dfa7550734 WatchSource:0}: Error finding container 26ea8bbfe8e9b6b619150c33a6fbb0e586f91bcb4397f12d0c1d09dfa7550734: Status 404 returned error can't find the container with id 26ea8bbfe8e9b6b619150c33a6fbb0e586f91bcb4397f12d0c1d09dfa7550734
Sep 17 18:42:28.280013 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:28.279979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" event={"ID":"8c7b6c22-dea2-47ce-8e1b-7fce54053f92","Type":"ContainerStarted","Data":"26ea8bbfe8e9b6b619150c33a6fbb0e586f91bcb4397f12d0c1d09dfa7550734"}
Sep 17 18:42:29.285885 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:29.285849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" event={"ID":"8c7b6c22-dea2-47ce-8e1b-7fce54053f92","Type":"ContainerStarted","Data":"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"}
Sep 17 18:42:29.286365 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:29.285927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:29.309913 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:29.309865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podStartSLOduration=2.309850992 podStartE2EDuration="2.309850992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:29.309652997 +0000 UTC m=+791.859876883" watchObservedRunningTime="2026-09-17 18:42:29.309850992 +0000 UTC m=+791.860074869"
Sep 17 18:42:30.144758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:30.144720    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:42:32.200621 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:32.200582    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:32.406846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:32.406808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:33.293338 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.293314    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:42:33.301199 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.301175    2559 generic.go:356] "Generic (PLEG): container finished" podID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerID="922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7" exitCode=0
Sep 17 18:42:33.301311 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.301212    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"
Sep 17 18:42:33.301311 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.301213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" event={"ID":"6a04d9a2-1eea-41ba-8a29-73a61d75a891","Type":"ContainerDied","Data":"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"}
Sep 17 18:42:33.301386 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.301348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8" event={"ID":"6a04d9a2-1eea-41ba-8a29-73a61d75a891","Type":"ContainerDied","Data":"fe5c915134400f2bce94aae69ad1fe5efced91dbc974ef86a57af0b74e7b293f"}
Sep 17 18:42:33.301386 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.301375    2559 scope.go:117] "RemoveContainer" containerID="922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"
Sep 17 18:42:33.309001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.308984    2559 scope.go:117] "RemoveContainer" containerID="922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"
Sep 17 18:42:33.309266 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:42:33.309249    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7\": container with ID starting with 922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7 not found: ID does not exist" containerID="922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"
Sep 17 18:42:33.309311 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.309273    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7"} err="failed to get container status \"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7\": rpc error: code = NotFound desc = could not find container \"922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7\": container with ID starting with 922bccd5b2fec5d925bfb2afc8e0b342ea862229ab45a757df1398b33484e1b7 not found: ID does not exist"
Sep 17 18:42:33.325703 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.325683    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls\") pod \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") "
Sep 17 18:42:33.325815 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.325716    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle\") pod \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\" (UID: \"6a04d9a2-1eea-41ba-8a29-73a61d75a891\") "
Sep 17 18:42:33.326118 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.326096    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6a04d9a2-1eea-41ba-8a29-73a61d75a891" (UID: "6a04d9a2-1eea-41ba-8a29-73a61d75a891"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:33.327654 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.327634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a04d9a2-1eea-41ba-8a29-73a61d75a891" (UID: "6a04d9a2-1eea-41ba-8a29-73a61d75a891"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:33.427041 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.426959    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a04d9a2-1eea-41ba-8a29-73a61d75a891-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:42:33.427041 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.426987    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a04d9a2-1eea-41ba-8a29-73a61d75a891-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:42:33.624582 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.624548    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:42:33.629728 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.629696    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f687ffc4-z65q8"]
Sep 17 18:42:33.940526 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:33.940493    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" path="/var/lib/kubelet/pods/6a04d9a2-1eea-41ba-8a29-73a61d75a891/volumes"
Sep 17 18:42:35.294934 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:35.294905    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:42:42.200704 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:42.200621    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 18:42:52.200675 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:42:52.200647    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:43:03.390322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.390281    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:43:03.390898 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.390852    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer"
Sep 17 18:43:03.390898 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.390872    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer"
Sep 17 18:43:03.391024 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.390969    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a04d9a2-1eea-41ba-8a29-73a61d75a891" containerName="model-chainer"
Sep 17 18:43:03.394038 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.394018    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.396351 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.396313    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-65928-kube-rbac-proxy-sar-config\""
Sep 17 18:43:03.396494 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.396438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-65928-serving-cert\""
Sep 17 18:43:03.396622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.396544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:03.405188 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.405168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.405283 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.405201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.407544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.407520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:43:03.506107 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.506068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.506107 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.506106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.506686 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.506668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.508385 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.508365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls\") pod \"sequence-graph-65928-767fdc487-v7j4m\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") " pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.518619 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.518596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:03.704140 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.704060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:03.981722 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:03.981642    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:43:03.984865 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:43:03.984820    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf487c4_deeb_47b3_b76e_cba9fa14275d.slice/crio-82e734cee30cf3d42730ec85796add46b7595f09d34928fed58eba4ba2a5a89a WatchSource:0}: Error finding container 82e734cee30cf3d42730ec85796add46b7595f09d34928fed58eba4ba2a5a89a: Status 404 returned error can't find the container with id 82e734cee30cf3d42730ec85796add46b7595f09d34928fed58eba4ba2a5a89a
Sep 17 18:43:04.428819 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:04.428782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" event={"ID":"2cf487c4-deeb-47b3-b76e-cba9fa14275d","Type":"ContainerStarted","Data":"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"}
Sep 17 18:43:04.429212 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:04.428859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" event={"ID":"2cf487c4-deeb-47b3-b76e-cba9fa14275d","Type":"ContainerStarted","Data":"82e734cee30cf3d42730ec85796add46b7595f09d34928fed58eba4ba2a5a89a"}
Sep 17 18:43:04.429212 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:04.428887    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:43:04.446344 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:04.446287    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podStartSLOduration=1.446266533 podStartE2EDuration="1.446266533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:04.446174366 +0000 UTC m=+826.996398253" watchObservedRunningTime="2026-09-17 18:43:04.446266533 +0000 UTC m=+826.996490418"
Sep 17 18:43:10.437943 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:43:10.437915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:44:17.869286 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:44:17.869212    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:44:17.872004 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:44:17.871983    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:46:15.699607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:46:15.699576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:30.606878 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:47:30.606779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:48.708580 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:48:48.708491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:17.895029 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:49:17.895000    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:49:17.898543 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:49:17.898516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:50:36.298642 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:36.298559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:41.704444 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.704411    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:50:41.705000 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.704630    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" containerID="cri-o://3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112" gracePeriod=30
Sep 17 18:50:41.734122 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.734087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:41.871055 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.871023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:41.938936 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.938899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:41.987455 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.987374    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:50:41.987622 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.987597    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container" containerID="cri-o://ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09" gracePeriod=30
Sep 17 18:50:41.987744 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:41.987702    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kube-rbac-proxy" containerID="cri-o://79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a" gracePeriod=30
Sep 17 18:50:42.042061 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.042025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:42.089240 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.089210    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:50:42.092751 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.092731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.095258 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.095231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1827-predictor-serving-cert\""
Sep 17 18:50:42.095404 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.095383    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1827-kube-rbac-proxy-sar-config\""
Sep 17 18:50:42.103971 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.103948    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:50:42.124085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.124057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvw8\" (UniqueName: \"kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.124232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.124122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.124232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.124152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.126818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.126785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:42.191589 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.191558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:42.225514 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.225478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvw8\" (UniqueName: \"kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.225703 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.225543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.225703 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.225577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.226253 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.226233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.228032 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.228014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.236524 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.236505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvw8\" (UniqueName: \"kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8\") pod \"error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") " pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.300247 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.300217    2559 generic.go:356] "Generic (PLEG): container finished" podID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerID="79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a" exitCode=2
Sep 17 18:50:42.300418 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.300291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerDied","Data":"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"}
Sep 17 18:50:42.405742 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.405701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:42.681786 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.681700    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:50:42.684693 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:50:42.684659    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bcbd63_731b_4fdb_bfb0_3f63e93ca912.slice/crio-7e2b11f912f4cad9b2546a6b58499ba39197469d06609e5507833eeb4b7acde3 WatchSource:0}: Error finding container 7e2b11f912f4cad9b2546a6b58499ba39197469d06609e5507833eeb4b7acde3: Status 404 returned error can't find the container with id 7e2b11f912f4cad9b2546a6b58499ba39197469d06609e5507833eeb4b7acde3
Sep 17 18:50:42.711060 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:42.711031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:43.305822 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.305784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerStarted","Data":"4d0c701ce2eae4c5a53ea64ffcc5ac6eb2720e63bb6fc313e9e28c7dc273da4e"}
Sep 17 18:50:43.306004 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.305868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerStarted","Data":"470993eef39e804d693e60f4a116b8321cfe63f0202c5539feefb4de71e6f446"}
Sep 17 18:50:43.306004 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.305900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerStarted","Data":"7e2b11f912f4cad9b2546a6b58499ba39197469d06609e5507833eeb4b7acde3"}
Sep 17 18:50:43.306004 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.305930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:43.306004 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.305958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:43.307513 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.307484    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:50:43.324437 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:43.324389    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podStartSLOduration=1.324371087 podStartE2EDuration="1.324371087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:43.323802172 +0000 UTC m=+1285.874026069" watchObservedRunningTime="2026-09-17 18:50:43.324371087 +0000 UTC m=+1285.874594977"
Sep 17 18:50:44.311104 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:44.311069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:50:45.240909 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.240878    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:50:45.251160 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.251136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z87ct\" (UniqueName: \"kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct\") pod \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") "
Sep 17 18:50:45.251277 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.251193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") pod \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") "
Sep 17 18:50:45.251277 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.251241    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config\") pod \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\" (UID: \"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb\") "
Sep 17 18:50:45.251601 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.251578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" (UID: "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:45.253151 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.253130    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct" (OuterVolumeSpecName: "kube-api-access-z87ct") pod "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" (UID: "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb"). InnerVolumeSpecName "kube-api-access-z87ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:45.253297 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.253274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" (UID: "dcd5d47d-4201-4f9d-ba5b-2220c662ebeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:45.294522 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.294482    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:50:45.317267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.317192    2559 generic.go:356] "Generic (PLEG): container finished" podID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerID="ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09" exitCode=0
Sep 17 18:50:45.317267 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.317240    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"
Sep 17 18:50:45.317697 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.317243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerDied","Data":"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"}
Sep 17 18:50:45.317697 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.317365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm" event={"ID":"dcd5d47d-4201-4f9d-ba5b-2220c662ebeb","Type":"ContainerDied","Data":"549f392d446f1824ba9079edbf7ca51b2bc8f0697878b45a594a8da92f93d6c1"}
Sep 17 18:50:45.317697 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.317391    2559 scope.go:117] "RemoveContainer" containerID="79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"
Sep 17 18:50:45.325994 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.325977    2559 scope.go:117] "RemoveContainer" containerID="ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"
Sep 17 18:50:45.333315 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.333293    2559 scope.go:117] "RemoveContainer" containerID="79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"
Sep 17 18:50:45.333595 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:50:45.333579    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a\": container with ID starting with 79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a not found: ID does not exist" containerID="79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"
Sep 17 18:50:45.333640 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.333604    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a"} err="failed to get container status \"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a\": rpc error: code = NotFound desc = could not find container \"79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a\": container with ID starting with 79489fda1106dc7ff9ec80e43d431a3eb95fbfb606752b4ef6a4b823ff32da3a not found: ID does not exist"
Sep 17 18:50:45.333640 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.333619    2559 scope.go:117] "RemoveContainer" containerID="ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"
Sep 17 18:50:45.334247 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:50:45.333817    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09\": container with ID starting with ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09 not found: ID does not exist" containerID="ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"
Sep 17 18:50:45.334247 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.333854    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09"} err="failed to get container status \"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09\": rpc error: code = NotFound desc = could not find container \"ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09\": container with ID starting with ca7ca6549db0b09f0401fbc1dfe97968742030f9b89c43743c4a4f3e5dc12c09 not found: ID does not exist"
Sep 17 18:50:45.338981 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.338958    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:50:45.342738 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.342715    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5f0c-predictor-78b5f64bc-74flm"]
Sep 17 18:50:45.351942 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.351924    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.351998 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.351945    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.351998 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.351956    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87ct\" (UniqueName: \"kubernetes.io/projected/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb-kube-api-access-z87ct\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.940528 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:45.940496    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" path="/var/lib/kubelet/pods/dcd5d47d-4201-4f9d-ba5b-2220c662ebeb/volumes"
Sep 17 18:50:49.316009 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:49.315979    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:50:49.316544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:49.316459    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:50:50.294031 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:50.293993    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:50:55.293372 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:55.293334    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:50:55.293769 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:55.293465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:50:59.316751 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:50:59.316710    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:51:00.293710 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:00.293671    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:05.293378 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:05.293337    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:09.316946 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:09.316903    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:51:10.293229 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:10.293189    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:12.363595 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.363564    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:51:12.436033 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.435998    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerID="3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112" exitCode=0
Sep 17 18:51:12.436213 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.436040    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"
Sep 17 18:51:12.436213 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.436058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" event={"ID":"8c7b6c22-dea2-47ce-8e1b-7fce54053f92","Type":"ContainerDied","Data":"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"}
Sep 17 18:51:12.436213 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.436097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz" event={"ID":"8c7b6c22-dea2-47ce-8e1b-7fce54053f92","Type":"ContainerDied","Data":"26ea8bbfe8e9b6b619150c33a6fbb0e586f91bcb4397f12d0c1d09dfa7550734"}
Sep 17 18:51:12.436213 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.436118    2559 scope.go:117] "RemoveContainer" containerID="3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"
Sep 17 18:51:12.444034 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.444013    2559 scope.go:117] "RemoveContainer" containerID="3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"
Sep 17 18:51:12.444313 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:12.444291    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112\": container with ID starting with 3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112 not found: ID does not exist" containerID="3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"
Sep 17 18:51:12.444377 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.444326    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112"} err="failed to get container status \"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112\": rpc error: code = NotFound desc = could not find container \"3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112\": container with ID starting with 3747ed05ef80a6a1ecaf864299f01e9d7e71c68f37efdc2f400ea7c0f9df2112 not found: ID does not exist"
Sep 17 18:51:12.484732 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.484651    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") pod \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") "
Sep 17 18:51:12.484884 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.484743    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle\") pod \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\" (UID: \"8c7b6c22-dea2-47ce-8e1b-7fce54053f92\") "
Sep 17 18:51:12.485114 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.485090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8c7b6c22-dea2-47ce-8e1b-7fce54053f92" (UID: "8c7b6c22-dea2-47ce-8e1b-7fce54053f92"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:12.486716 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.486700    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c7b6c22-dea2-47ce-8e1b-7fce54053f92" (UID: "8c7b6c22-dea2-47ce-8e1b-7fce54053f92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:12.586143 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.586105    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:12.586143 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.586137    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7b6c22-dea2-47ce-8e1b-7fce54053f92-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:12.760413 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.760371    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:51:12.769565 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:12.766648    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c5f0c-844f475959-vnjtz"]
Sep 17 18:51:13.941014 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:13.940976    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" path="/var/lib/kubelet/pods/8c7b6c22-dea2-47ce-8e1b-7fce54053f92/volumes"
Sep 17 18:51:18.063459 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.063421    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:51:18.063869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.063630    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" containerID="cri-o://8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18" gracePeriod=30
Sep 17 18:51:18.081498 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.081462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:18.273869 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.273808    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:51:18.274060 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.274037    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container" containerID="cri-o://8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e" gracePeriod=30
Sep 17 18:51:18.274128 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.274086    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kube-rbac-proxy" containerID="cri-o://7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0" gracePeriod=30
Sep 17 18:51:18.462700 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.462663    2559 generic.go:356] "Generic (PLEG): container finished" podID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerID="7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0" exitCode=2
Sep 17 18:51:18.462894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.462732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerDied","Data":"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"}
Sep 17 18:51:18.553821 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.553713    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:51:18.554079 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554065    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c"
Sep 17 18:51:18.554123 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554081    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c"
Sep 17 18:51:18.554123 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554097    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kube-rbac-proxy"
Sep 17 18:51:18.554123 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554102    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kube-rbac-proxy"
Sep 17 18:51:18.554123 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554119    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container"
Sep 17 18:51:18.554249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554125    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container"
Sep 17 18:51:18.554249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554182    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7b6c22-dea2-47ce-8e1b-7fce54053f92" containerName="switch-graph-c5f0c"
Sep 17 18:51:18.554249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554189    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kserve-container"
Sep 17 18:51:18.554249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.554195    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd5d47d-4201-4f9d-ba5b-2220c662ebeb" containerName="kube-rbac-proxy"
Sep 17 18:51:18.556954 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.556927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.560015 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.559990    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31957-predictor-serving-cert\""
Sep 17 18:51:18.560154 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.559993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31957-kube-rbac-proxy-sar-config\""
Sep 17 18:51:18.579702 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.579669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:51:18.639251 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.639211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jv5\" (UniqueName: \"kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.639251 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.639257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.639478 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.639283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.740023 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.739987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jv5\" (UniqueName: \"kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.740023 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.740028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.740236 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.740061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.740236 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:18.740151    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-31957-predictor-serving-cert: secret "error-404-isvc-31957-predictor-serving-cert" not found
Sep 17 18:51:18.740236 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:18.740228    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls podName:121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e nodeName:}" failed. No retries permitted until 2026-09-17 18:51:19.240205805 +0000 UTC m=+1321.790429687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls") pod "error-404-isvc-31957-predictor-5657f8887-srjcj" (UID: "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e") : secret "error-404-isvc-31957-predictor-serving-cert" not found
Sep 17 18:51:18.740792 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.740771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:18.750279 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:18.750244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jv5\" (UniqueName: \"kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:19.244449 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.244404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:19.246847 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.246804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") pod \"error-404-isvc-31957-predictor-5657f8887-srjcj\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") " pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:19.316467 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.316426    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:51:19.466846 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.466792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:19.656282 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.656249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:19.756801 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:19.756770    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:51:19.758605 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:51:19.758581    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121d13e0_d2dd_4c59_8c3e_1bf8fbc86f4e.slice/crio-205e1915f111845aa421c7a1bd931e766a2f699a11ad804a923fdef726845728 WatchSource:0}: Error finding container 205e1915f111845aa421c7a1bd931e766a2f699a11ad804a923fdef726845728: Status 404 returned error can't find the container with id 205e1915f111845aa421c7a1bd931e766a2f699a11ad804a923fdef726845728
Sep 17 18:51:20.436599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.436557    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:20.472308 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.472277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerStarted","Data":"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"}
Sep 17 18:51:20.472483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.472341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerStarted","Data":"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"}
Sep 17 18:51:20.472483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.472370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerStarted","Data":"205e1915f111845aa421c7a1bd931e766a2f699a11ad804a923fdef726845728"}
Sep 17 18:51:20.472483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.472413    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:20.472483 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.472441    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:20.473965 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.473937    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:20.491223 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:20.491170    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podStartSLOduration=2.491154381 podStartE2EDuration="2.491154381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:20.489932661 +0000 UTC m=+1323.040156549" watchObservedRunningTime="2026-09-17 18:51:20.491154381 +0000 UTC m=+1323.041378270"
Sep 17 18:51:21.476937 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.476900    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:21.624815 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.624784    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:51:21.767472 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.767430    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config\") pod \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") "
Sep 17 18:51:21.767638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.767520    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6psqm\" (UniqueName: \"kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm\") pod \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") "
Sep 17 18:51:21.767638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.767590    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls\") pod \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\" (UID: \"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea\") "
Sep 17 18:51:21.767871 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.767817    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" (UID: "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:21.769544 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.769510    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm" (OuterVolumeSpecName: "kube-api-access-6psqm") pod "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" (UID: "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea"). InnerVolumeSpecName "kube-api-access-6psqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:21.769644 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.769599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" (UID: "a7ad6913-e42a-4ee3-8e5e-804a529bd5ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:21.868941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.868904    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:21.868941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.868935    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:21.868941 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:21.868945    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6psqm\" (UniqueName: \"kubernetes.io/projected/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea-kube-api-access-6psqm\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:22.482730 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.482696    2559 generic.go:356] "Generic (PLEG): container finished" podID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerID="8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e" exitCode=0
Sep 17 18:51:22.483159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.482745    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"
Sep 17 18:51:22.483159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.482765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerDied","Data":"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"}
Sep 17 18:51:22.483159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.482858    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9" event={"ID":"a7ad6913-e42a-4ee3-8e5e-804a529bd5ea","Type":"ContainerDied","Data":"1682080326ed851a6eb3e78ac79962f7a766de224958e6ebe4218127cefc4960"}
Sep 17 18:51:22.483159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.482899    2559 scope.go:117] "RemoveContainer" containerID="7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"
Sep 17 18:51:22.491328 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.491100    2559 scope.go:117] "RemoveContainer" containerID="8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"
Sep 17 18:51:22.498376 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.498357    2559 scope.go:117] "RemoveContainer" containerID="7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"
Sep 17 18:51:22.498659 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:22.498640    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0\": container with ID starting with 7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0 not found: ID does not exist" containerID="7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"
Sep 17 18:51:22.498736 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.498669    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0"} err="failed to get container status \"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0\": rpc error: code = NotFound desc = could not find container \"7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0\": container with ID starting with 7270fdb2ce60673e5a166510cb97e639d78ee67e7585ca72ab59137a6deeb5f0 not found: ID does not exist"
Sep 17 18:51:22.498736 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.498685    2559 scope.go:117] "RemoveContainer" containerID="8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"
Sep 17 18:51:22.498989 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:22.498969    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e\": container with ID starting with 8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e not found: ID does not exist" containerID="8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"
Sep 17 18:51:22.499064 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.498995    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e"} err="failed to get container status \"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e\": rpc error: code = NotFound desc = could not find container \"8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e\": container with ID starting with 8be9ab07623e6d45762b7418e595fd10ade43e84683776deee48b07b243e0a1e not found: ID does not exist"
Sep 17 18:51:22.501971 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.501951    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:51:22.508942 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:22.508921    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65928-predictor-6b77457dbf-g7kr9"]
Sep 17 18:51:23.941390 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:23.941353    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" path="/var/lib/kubelet/pods/a7ad6913-e42a-4ee3-8e5e-804a529bd5ea/volumes"
Sep 17 18:51:25.436553 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:25.436503    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:26.481407 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:26.481373    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:51:26.481931 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:26.481903    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:29.317007 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:29.316977    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:51:30.436688 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:30.436648    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:30.437114 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:30.436798    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:51:35.436719 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:35.436675    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:36.482224 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:36.482183    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:40.436999 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:40.436913    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:41.976326 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976294    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:51:41.976926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kube-rbac-proxy"
Sep 17 18:51:41.976926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976794    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kube-rbac-proxy"
Sep 17 18:51:41.976926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976821    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container"
Sep 17 18:51:41.976926 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976847    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container"
Sep 17 18:51:41.977155 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976948    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kserve-container"
Sep 17 18:51:41.977155 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.976963    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ad6913-e42a-4ee3-8e5e-804a529bd5ea" containerName="kube-rbac-proxy"
Sep 17 18:51:41.980973 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.980950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:41.984819 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.984794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c1827-serving-cert\""
Sep 17 18:51:41.985147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.985123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c1827-kube-rbac-proxy-sar-config\""
Sep 17 18:51:41.997047 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.997024    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:51:41.998050 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:41.998024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:42.038570 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.038536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.038738 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.038577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.139140 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.139107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.139140 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.139141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.139351 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:42.139251    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-c1827-serving-cert: secret "ensemble-graph-c1827-serving-cert" not found
Sep 17 18:51:42.139351 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:42.139311    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls podName:a6b193c5-4627-45ed-be0e-d8ad5bf22ee2 nodeName:}" failed. No retries permitted until 2026-09-17 18:51:42.639295661 +0000 UTC m=+1345.189519527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls") pod "ensemble-graph-c1827-745964469b-cskkx" (UID: "a6b193c5-4627-45ed-be0e-d8ad5bf22ee2") : secret "ensemble-graph-c1827-serving-cert" not found
Sep 17 18:51:42.139726 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.139704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.356555 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.356513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:42.641856 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.641752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.644156 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.644136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") pod \"ensemble-graph-c1827-745964469b-cskkx\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") " pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:42.891382 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:42.891332    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:43.161113 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:43.160999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:51:43.163865 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:51:43.163801    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b193c5_4627_45ed_be0e_d8ad5bf22ee2.slice/crio-c24bbaa9d5f2891ff39ede7e981134267e905d5af6278ab4a6f3199a5da7b17c WatchSource:0}: Error finding container c24bbaa9d5f2891ff39ede7e981134267e905d5af6278ab4a6f3199a5da7b17c: Status 404 returned error can't find the container with id c24bbaa9d5f2891ff39ede7e981134267e905d5af6278ab4a6f3199a5da7b17c
Sep 17 18:51:43.582101 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:43.582065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" event={"ID":"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2","Type":"ContainerStarted","Data":"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"}
Sep 17 18:51:43.582277 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:43.582118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" event={"ID":"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2","Type":"ContainerStarted","Data":"c24bbaa9d5f2891ff39ede7e981134267e905d5af6278ab4a6f3199a5da7b17c"}
Sep 17 18:51:43.582277 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:43.582165    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:43.598375 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:43.598323    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podStartSLOduration=2.598306704 podStartE2EDuration="2.598306704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:43.597005918 +0000 UTC m=+1346.147229808" watchObservedRunningTime="2026-09-17 18:51:43.598306704 +0000 UTC m=+1346.148530590"
Sep 17 18:51:45.437234 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:45.437196    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:46.482199 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:46.482155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:48.206249 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.206222    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:51:48.284845 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.284793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls\") pod \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") "
Sep 17 18:51:48.284845 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.284852    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle\") pod \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\" (UID: \"2cf487c4-deeb-47b3-b76e-cba9fa14275d\") "
Sep 17 18:51:48.285232 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.285203    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2cf487c4-deeb-47b3-b76e-cba9fa14275d" (UID: "2cf487c4-deeb-47b3-b76e-cba9fa14275d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:48.286938 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.286916    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cf487c4-deeb-47b3-b76e-cba9fa14275d" (UID: "2cf487c4-deeb-47b3-b76e-cba9fa14275d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:48.386002 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.385910    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cf487c4-deeb-47b3-b76e-cba9fa14275d-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:48.386002 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.385947    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf487c4-deeb-47b3-b76e-cba9fa14275d-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:48.604147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.604109    2559 generic.go:356] "Generic (PLEG): container finished" podID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerID="8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18" exitCode=0
Sep 17 18:51:48.604322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.604157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" event={"ID":"2cf487c4-deeb-47b3-b76e-cba9fa14275d","Type":"ContainerDied","Data":"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"}
Sep 17 18:51:48.604322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.604179    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"
Sep 17 18:51:48.604322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.604229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m" event={"ID":"2cf487c4-deeb-47b3-b76e-cba9fa14275d","Type":"ContainerDied","Data":"82e734cee30cf3d42730ec85796add46b7595f09d34928fed58eba4ba2a5a89a"}
Sep 17 18:51:48.604322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.604257    2559 scope.go:117] "RemoveContainer" containerID="8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"
Sep 17 18:51:48.614067 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.614044    2559 scope.go:117] "RemoveContainer" containerID="8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"
Sep 17 18:51:48.614409 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:51:48.614387    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18\": container with ID starting with 8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18 not found: ID does not exist" containerID="8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"
Sep 17 18:51:48.614517 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.614418    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18"} err="failed to get container status \"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18\": rpc error: code = NotFound desc = could not find container \"8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18\": container with ID starting with 8841cbffd10a4ae5fc4896b0119a06f16ffbb13ac34daf33b8741bb109e13f18 not found: ID does not exist"
Sep 17 18:51:48.625039 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.625009    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:51:48.628743 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:48.628711    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-65928-767fdc487-v7j4m"]
Sep 17 18:51:49.590723 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:49.590692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:51:49.941052 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:49.940959    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" path="/var/lib/kubelet/pods/2cf487c4-deeb-47b3-b76e-cba9fa14275d/volumes"
Sep 17 18:51:52.039511 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.039482    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:51:52.039902 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.039654    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" containerID="cri-o://ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3" gracePeriod=30
Sep 17 18:51:52.073636 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.073603    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.196858 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.196808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.243231 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.243197    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:51:52.243511 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.243480    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container" containerID="cri-o://470993eef39e804d693e60f4a116b8321cfe63f0202c5539feefb4de71e6f446" gracePeriod=30
Sep 17 18:51:52.243646 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.243551    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kube-rbac-proxy" containerID="cri-o://4d0c701ce2eae4c5a53ea64ffcc5ac6eb2720e63bb6fc313e9e28c7dc273da4e" gracePeriod=30
Sep 17 18:51:52.281045 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.281012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.392520 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.392487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.411367 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.411340    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 18:51:52.411668 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.411655    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928"
Sep 17 18:51:52.411668 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.411669    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928"
Sep 17 18:51:52.411803 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.411745    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf487c4-deeb-47b3-b76e-cba9fa14275d" containerName="sequence-graph-65928"
Sep 17 18:51:52.414549 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.414533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.417172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.417152    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-10598-kube-rbac-proxy-sar-config\""
Sep 17 18:51:52.417396 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.417370    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-10598-predictor-serving-cert\""
Sep 17 18:51:52.432986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.432965    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 18:51:52.512173 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.512136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.512173 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.512172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.512398 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.512236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l8x\" (UniqueName: \"kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.612969 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.612872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l8x\" (UniqueName: \"kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.612969 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.612951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.612969 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.612969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.613606 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.613581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.614631 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.614606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.615628 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.615609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.621492 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.621469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l8x\" (UniqueName: \"kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x\") pod \"error-404-isvc-10598-predictor-577cdb5f5f-xf8dr\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") " pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:52.624047 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.624024    2559 generic.go:356] "Generic (PLEG): container finished" podID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerID="4d0c701ce2eae4c5a53ea64ffcc5ac6eb2720e63bb6fc313e9e28c7dc273da4e" exitCode=2
Sep 17 18:51:52.624139 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.624087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerDied","Data":"4d0c701ce2eae4c5a53ea64ffcc5ac6eb2720e63bb6fc313e9e28c7dc273da4e"}
Sep 17 18:51:52.714788 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.714753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:52.724779 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:52.724741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:53.001901 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.001876    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 18:51:53.004282 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:51:53.004257    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e8da0e_1bf4_434c_b658_df2c140dffc2.slice/crio-126d3b0620b05da7e72e9930b50ab80a983cbe70cf98979ca8e567948dc104cc WatchSource:0}: Error finding container 126d3b0620b05da7e72e9930b50ab80a983cbe70cf98979ca8e567948dc104cc: Status 404 returned error can't find the container with id 126d3b0620b05da7e72e9930b50ab80a983cbe70cf98979ca8e567948dc104cc
Sep 17 18:51:53.630894 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.630856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerStarted","Data":"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"}
Sep 17 18:51:53.631339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.630930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:53.631339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.630960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerStarted","Data":"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"}
Sep 17 18:51:53.631339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.630992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:53.631339 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.631016    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerStarted","Data":"126d3b0620b05da7e72e9930b50ab80a983cbe70cf98979ca8e567948dc104cc"}
Sep 17 18:51:53.632621 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.632595    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:51:53.653607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:53.653533    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podStartSLOduration=1.653509969 podStartE2EDuration="1.653509969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:53.650484347 +0000 UTC m=+1356.200708251" watchObservedRunningTime="2026-09-17 18:51:53.653509969 +0000 UTC m=+1356.203733859"
Sep 17 18:51:54.312322 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:54.312273    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 18:51:54.589766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:54.589668    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:54.637042 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:54.637003    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:51:55.643531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.643497    2559 generic.go:356] "Generic (PLEG): container finished" podID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerID="470993eef39e804d693e60f4a116b8321cfe63f0202c5539feefb4de71e6f446" exitCode=0
Sep 17 18:51:55.644029 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.643575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerDied","Data":"470993eef39e804d693e60f4a116b8321cfe63f0202c5539feefb4de71e6f446"}
Sep 17 18:51:55.696085 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.696059    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:51:55.738432 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.738356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls\") pod \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") "
Sep 17 18:51:55.738432 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.738407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config\") pod \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") "
Sep 17 18:51:55.738664 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.738441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stvw8\" (UniqueName: \"kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8\") pod \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\" (UID: \"00bcbd63-731b-4fdb-bfb0-3f63e93ca912\") "
Sep 17 18:51:55.738845 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.738781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00bcbd63-731b-4fdb-bfb0-3f63e93ca912" (UID: "00bcbd63-731b-4fdb-bfb0-3f63e93ca912"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:55.740523 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.740493    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00bcbd63-731b-4fdb-bfb0-3f63e93ca912" (UID: "00bcbd63-731b-4fdb-bfb0-3f63e93ca912"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:55.740610 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.740561    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8" (OuterVolumeSpecName: "kube-api-access-stvw8") pod "00bcbd63-731b-4fdb-bfb0-3f63e93ca912" (UID: "00bcbd63-731b-4fdb-bfb0-3f63e93ca912"). InnerVolumeSpecName "kube-api-access-stvw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:55.839112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.839076    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:55.839112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.839108    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:55.839112 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:55.839119    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvw8\" (UniqueName: \"kubernetes.io/projected/00bcbd63-731b-4fdb-bfb0-3f63e93ca912-kube-api-access-stvw8\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:51:56.482744 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.482703    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:51:56.650082 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.650053    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"
Sep 17 18:51:56.650501 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.650050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm" event={"ID":"00bcbd63-731b-4fdb-bfb0-3f63e93ca912","Type":"ContainerDied","Data":"7e2b11f912f4cad9b2546a6b58499ba39197469d06609e5507833eeb4b7acde3"}
Sep 17 18:51:56.650501 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.650222    2559 scope.go:117] "RemoveContainer" containerID="4d0c701ce2eae4c5a53ea64ffcc5ac6eb2720e63bb6fc313e9e28c7dc273da4e"
Sep 17 18:51:56.658637 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.658618    2559 scope.go:117] "RemoveContainer" containerID="470993eef39e804d693e60f4a116b8321cfe63f0202c5539feefb4de71e6f446"
Sep 17 18:51:56.683546 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.683517    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:51:56.693130 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:56.693102    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1827-predictor-5fccd85d8c-9dtwm"]
Sep 17 18:51:57.940540 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:57.940504    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" path="/var/lib/kubelet/pods/00bcbd63-731b-4fdb-bfb0-3f63e93ca912/volumes"
Sep 17 18:51:59.589722 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:59.589680    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:51:59.641445 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:59.641413    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:51:59.642057 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:51:59.642028    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:52:04.589751 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:04.589709    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:04.590144 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:04.589813    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:52:06.486638 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:06.486599    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:52:09.589881 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:09.589824    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:09.642813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:09.642779    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:52:14.589818 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:14.589770    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:18.275797 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.275761    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:18.276174 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276103    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kube-rbac-proxy"
Sep 17 18:52:18.276174 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276115    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kube-rbac-proxy"
Sep 17 18:52:18.276174 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276138    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container"
Sep 17 18:52:18.276174 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276143    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container"
Sep 17 18:52:18.276305 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276200    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kube-rbac-proxy"
Sep 17 18:52:18.276305 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.276209    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bcbd63-731b-4fdb-bfb0-3f63e93ca912" containerName="kserve-container"
Sep 17 18:52:18.280471 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.280456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.283012 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.282991    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-31957-serving-cert\""
Sep 17 18:52:18.283109 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.282993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-31957-kube-rbac-proxy-sar-config\""
Sep 17 18:52:18.289173 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.289152    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:18.296386 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.296366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:18.324534 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.324504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.324690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.324553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.425857 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.425782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.425857 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.425864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.426074 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:18.425966    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-31957-serving-cert: secret "sequence-graph-31957-serving-cert" not found
Sep 17 18:52:18.426074 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:18.426022    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls podName:a3134244-4972-426b-9711-39f265dc6585 nodeName:}" failed. No retries permitted until 2026-09-17 18:52:18.926006011 +0000 UTC m=+1381.476229876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls") pod "sequence-graph-31957-5d489cff66-4lgrj" (UID: "a3134244-4972-426b-9711-39f265dc6585") : secret "sequence-graph-31957-serving-cert" not found
Sep 17 18:52:18.426503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.426479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.644469 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.644378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:18.929809 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.929722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:18.932103 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:18.932085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") pod \"sequence-graph-31957-5d489cff66-4lgrj\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") " pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:19.191765 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.191685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:19.461466 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.461439    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:19.590021 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.589981    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:19.642822 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.642772    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:52:19.749310 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.749276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" event={"ID":"a3134244-4972-426b-9711-39f265dc6585","Type":"ContainerStarted","Data":"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"}
Sep 17 18:52:19.749516 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.749335    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" event={"ID":"a3134244-4972-426b-9711-39f265dc6585","Type":"ContainerStarted","Data":"60144fcc032f553f8a37d45ffa6e9126913614a68a833da36ddc7a0fb47ccb05"}
Sep 17 18:52:19.749516 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.749376    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:19.767381 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:19.767324    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podStartSLOduration=1.7673075219999999 podStartE2EDuration="1.767307522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:19.766385941 +0000 UTC m=+1382.316609829" watchObservedRunningTime="2026-09-17 18:52:19.767307522 +0000 UTC m=+1382.317531468"
Sep 17 18:52:22.181563 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.181541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:52:22.257690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.257660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") pod \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") "
Sep 17 18:52:22.257874 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.257759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle\") pod \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\" (UID: \"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2\") "
Sep 17 18:52:22.258058 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.258034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" (UID: "a6b193c5-4627-45ed-be0e-d8ad5bf22ee2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:22.258125 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.258099    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:22.259977 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.259951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" (UID: "a6b193c5-4627-45ed-be0e-d8ad5bf22ee2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:22.358684 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.358600    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:22.761762 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.761729    2559 generic.go:356] "Generic (PLEG): container finished" podID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerID="ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3" exitCode=0
Sep 17 18:52:22.761951 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.761774    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"
Sep 17 18:52:22.761951 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.761809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" event={"ID":"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2","Type":"ContainerDied","Data":"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"}
Sep 17 18:52:22.761951 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.761877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx" event={"ID":"a6b193c5-4627-45ed-be0e-d8ad5bf22ee2","Type":"ContainerDied","Data":"c24bbaa9d5f2891ff39ede7e981134267e905d5af6278ab4a6f3199a5da7b17c"}
Sep 17 18:52:22.761951 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.761908    2559 scope.go:117] "RemoveContainer" containerID="ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"
Sep 17 18:52:22.770159 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.770137    2559 scope.go:117] "RemoveContainer" containerID="ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"
Sep 17 18:52:22.770428 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:22.770407    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3\": container with ID starting with ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3 not found: ID does not exist" containerID="ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"
Sep 17 18:52:22.770479 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.770437    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3"} err="failed to get container status \"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3\": rpc error: code = NotFound desc = could not find container \"ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3\": container with ID starting with ebaa789b749b01301cdae9c28c55e5ea0b9dca9008bedd5e348e38b6b9df0ce3 not found: ID does not exist"
Sep 17 18:52:22.782647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.782624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:52:22.789224 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:22.789202    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1827-745964469b-cskkx"]
Sep 17 18:52:23.939986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:23.939956    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" path="/var/lib/kubelet/pods/a6b193c5-4627-45ed-be0e-d8ad5bf22ee2/volumes"
Sep 17 18:52:25.758357 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:25.758331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:28.357744 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.357709    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:28.358147 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.357905    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" containerID="cri-o://cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2" gracePeriod=30
Sep 17 18:52:28.388871 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.388840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.473447 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.473419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.485073 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.485034    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:52:28.485305 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.485281    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container" containerID="cri-o://bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508" gracePeriod=30
Sep 17 18:52:28.485384 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.485347    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kube-rbac-proxy" containerID="cri-o://3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e" gracePeriod=30
Sep 17 18:52:28.553939 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.553901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.648813 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.648696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.694102 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.694073    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 18:52:28.694402 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.694390    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827"
Sep 17 18:52:28.694447 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.694403    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827"
Sep 17 18:52:28.694481 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.694473    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b193c5-4627-45ed-be0e-d8ad5bf22ee2" containerName="ensemble-graph-c1827"
Sep 17 18:52:28.697439 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.697422    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.700134 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.700105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c4e90-predictor-serving-cert\""
Sep 17 18:52:28.700262 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.700119    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c4e90-kube-rbac-proxy-sar-config\""
Sep 17 18:52:28.714737 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.714708    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 18:52:28.729181 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.729142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.788126 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.788092    2559 generic.go:356] "Generic (PLEG): container finished" podID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerID="3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e" exitCode=2
Sep 17 18:52:28.788306 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.788125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerDied","Data":"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"}
Sep 17 18:52:28.801729 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.801696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.810843 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.810806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.810929 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.810895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.811005 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.810988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhqs\" (UniqueName: \"kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.911620 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.911533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhqs\" (UniqueName: \"kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.911620 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.911569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.911620 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.911597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.911922 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:28.911693    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-serving-cert: secret "error-404-isvc-c4e90-predictor-serving-cert" not found
Sep 17 18:52:28.911922 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:28.911752    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls podName:99a9d39b-0547-41c1-93a7-42ce6a71ea15 nodeName:}" failed. No retries permitted until 2026-09-17 18:52:29.411733578 +0000 UTC m=+1391.961957457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls") pod "error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" (UID: "99a9d39b-0547-41c1-93a7-42ce6a71ea15") : secret "error-404-isvc-c4e90-predictor-serving-cert" not found
Sep 17 18:52:28.912275 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.912254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:28.922591 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:28.922565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhqs\" (UniqueName: \"kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:29.026531 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.026499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:29.416102 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.416060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:29.418572 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.418545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") pod \"error-404-isvc-c4e90-predictor-555f9566bc-fnlzr\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") " pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:29.607034 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.606990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:29.642184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.642141    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:52:29.883600 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:29.883423    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 18:52:29.886459 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:52:29.886426    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a9d39b_0547_41c1_93a7_42ce6a71ea15.slice/crio-635f1c456d4f635eda9a20d70c1fa0d011376ea792502433d80bbb776bd6265d WatchSource:0}: Error finding container 635f1c456d4f635eda9a20d70c1fa0d011376ea792502433d80bbb776bd6265d: Status 404 returned error can't find the container with id 635f1c456d4f635eda9a20d70c1fa0d011376ea792502433d80bbb776bd6265d
Sep 17 18:52:30.757758 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.757720    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:30.798404 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.798371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerStarted","Data":"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"}
Sep 17 18:52:30.798617 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.798438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerStarted","Data":"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"}
Sep 17 18:52:30.798617 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.798471    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:30.798617 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.798499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerStarted","Data":"635f1c456d4f635eda9a20d70c1fa0d011376ea792502433d80bbb776bd6265d"}
Sep 17 18:52:30.818273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:30.818208    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podStartSLOduration=2.818189891 podStartE2EDuration="2.818189891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:30.817264126 +0000 UTC m=+1393.367488010" watchObservedRunningTime="2026-09-17 18:52:30.818189891 +0000 UTC m=+1393.368413777"
Sep 17 18:52:31.477438 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.477393    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:52:31.727308 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.727282    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:52:31.804204 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804171    2559 generic.go:356] "Generic (PLEG): container finished" podID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerID="bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508" exitCode=0
Sep 17 18:52:31.804647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerDied","Data":"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"}
Sep 17 18:52:31.804647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj" event={"ID":"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e","Type":"ContainerDied","Data":"205e1915f111845aa421c7a1bd931e766a2f699a11ad804a923fdef726845728"}
Sep 17 18:52:31.804647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804298    2559 scope.go:117] "RemoveContainer" containerID="3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"
Sep 17 18:52:31.804647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804217    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"
Sep 17 18:52:31.804647 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.804403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:31.805577 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.805551    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:52:31.812284 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.812265    2559 scope.go:117] "RemoveContainer" containerID="bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"
Sep 17 18:52:31.819401 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.819384    2559 scope.go:117] "RemoveContainer" containerID="3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"
Sep 17 18:52:31.819650 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:31.819622    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e\": container with ID starting with 3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e not found: ID does not exist" containerID="3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"
Sep 17 18:52:31.819705 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.819650    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e"} err="failed to get container status \"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e\": rpc error: code = NotFound desc = could not find container \"3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e\": container with ID starting with 3d2e196323fa8372adfa048704b02f45dfc90d63de31e4a18ee7369be0854c7e not found: ID does not exist"
Sep 17 18:52:31.819705 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.819671    2559 scope.go:117] "RemoveContainer" containerID="bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"
Sep 17 18:52:31.819951 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:31.819935    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508\": container with ID starting with bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508 not found: ID does not exist" containerID="bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"
Sep 17 18:52:31.820010 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.819954    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508"} err="failed to get container status \"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508\": rpc error: code = NotFound desc = could not find container \"bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508\": container with ID starting with bf9bbcd93ffdfb9428cff160302e4c734d263abb4ad0b280b3a54cc08eb19508 not found: ID does not exist"
Sep 17 18:52:31.839227 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.839207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config\") pod \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") "
Sep 17 18:52:31.839305 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.839272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5jv5\" (UniqueName: \"kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5\") pod \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") "
Sep 17 18:52:31.839343 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.839330    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") pod \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\" (UID: \"121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e\") "
Sep 17 18:52:31.839576 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.839553    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" (UID: "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:31.841273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.841242    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5" (OuterVolumeSpecName: "kube-api-access-w5jv5") pod "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" (UID: "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e"). InnerVolumeSpecName "kube-api-access-w5jv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:31.841273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.841260    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" (UID: "121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:31.940172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.940135    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:31.940172 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.940166    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:31.940377 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:31.940181    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5jv5\" (UniqueName: \"kubernetes.io/projected/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e-kube-api-access-w5jv5\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:32.132220 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:32.132135    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:52:32.135599 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:32.135574    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31957-predictor-5657f8887-srjcj"]
Sep 17 18:52:32.809086 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:32.809046    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:52:33.941019 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:33.940982    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" path="/var/lib/kubelet/pods/121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e/volumes"
Sep 17 18:52:35.757746 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:35.757699    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:37.813958 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:37.813927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:52:37.814371 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:37.814346    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:52:39.643215 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:39.643177    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 18:52:40.757276 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:40.757236    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:40.757708 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:40.757360    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:45.757794 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:45.757754    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:47.815082 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:47.815044    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:52:50.757603 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:50.757558    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:52.292212 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292177    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 18:52:52.292607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container"
Sep 17 18:52:52.292607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292524    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container"
Sep 17 18:52:52.292607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292534    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kube-rbac-proxy"
Sep 17 18:52:52.292607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292539    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kube-rbac-proxy"
Sep 17 18:52:52.292607 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292602    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kserve-container"
Sep 17 18:52:52.292795 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.292612    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="121d13e0-d2dd-4c59-8c3e-1bf8fbc86f4e" containerName="kube-rbac-proxy"
Sep 17 18:52:52.295584 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.295567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.298205 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.298182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-10598-serving-cert\""
Sep 17 18:52:52.298664 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.298646    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-10598-kube-rbac-proxy-sar-config\""
Sep 17 18:52:52.304781 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.304760    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 18:52:52.316873 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.316820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.316988 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.316950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.348710 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.348670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:52.417592 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.417558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.417766 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.417620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.418323 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.418300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.419986 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.419965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls\") pod \"ensemble-graph-10598-647cf68fb6-9lwpp\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") " pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.466409 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.466377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:52.606581 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.606493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:52.879984 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.879899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 18:52:52.882490 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:52:52.882461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e021e88_3fdb_4afb_b7ed_88bb95340dfa.slice/crio-a443176210aab6f4aa126592a97156aab13e8d50f258153e73103d7b2c51d239 WatchSource:0}: Error finding container a443176210aab6f4aa126592a97156aab13e8d50f258153e73103d7b2c51d239: Status 404 returned error can't find the container with id a443176210aab6f4aa126592a97156aab13e8d50f258153e73103d7b2c51d239
Sep 17 18:52:52.892810 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:52.892777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" event={"ID":"1e021e88-3fdb-4afb-b7ed-88bb95340dfa","Type":"ContainerStarted","Data":"a443176210aab6f4aa126592a97156aab13e8d50f258153e73103d7b2c51d239"}
Sep 17 18:52:53.904494 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:53.904445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" event={"ID":"1e021e88-3fdb-4afb-b7ed-88bb95340dfa","Type":"ContainerStarted","Data":"87d471c3ee3c32292498426ede6e8ecbcf0e015403c78308ed74b308d7c00f85"}
Sep 17 18:52:53.905021 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:53.904602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:53.923450 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:53.923384    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podStartSLOduration=1.923369262 podStartE2EDuration="1.923369262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:53.921882162 +0000 UTC m=+1416.472106049" watchObservedRunningTime="2026-09-17 18:52:53.923369262 +0000 UTC m=+1416.473593148"
Sep 17 18:52:55.757746 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:55.757705    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:52:57.815120 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:57.815079    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:52:58.502732 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.502708    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:58.576255 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.576219    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") pod \"a3134244-4972-426b-9711-39f265dc6585\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") "
Sep 17 18:52:58.576406 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.576308    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle\") pod \"a3134244-4972-426b-9711-39f265dc6585\" (UID: \"a3134244-4972-426b-9711-39f265dc6585\") "
Sep 17 18:52:58.576690 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.576655    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a3134244-4972-426b-9711-39f265dc6585" (UID: "a3134244-4972-426b-9711-39f265dc6585"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:58.578292 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.578269    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3134244-4972-426b-9711-39f265dc6585" (UID: "a3134244-4972-426b-9711-39f265dc6585"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:58.677372 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.677288    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3134244-4972-426b-9711-39f265dc6585-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:58.677372 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.677319    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3134244-4972-426b-9711-39f265dc6585-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 18:52:58.928327 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.928232    2559 generic.go:356] "Generic (PLEG): container finished" podID="a3134244-4972-426b-9711-39f265dc6585" containerID="cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2" exitCode=0
Sep 17 18:52:58.928327 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.928274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" event={"ID":"a3134244-4972-426b-9711-39f265dc6585","Type":"ContainerDied","Data":"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"}
Sep 17 18:52:58.928327 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.928324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj" event={"ID":"a3134244-4972-426b-9711-39f265dc6585","Type":"ContainerDied","Data":"60144fcc032f553f8a37d45ffa6e9126913614a68a833da36ddc7a0fb47ccb05"}
Sep 17 18:52:58.928896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.928345    2559 scope.go:117] "RemoveContainer" containerID="cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"
Sep 17 18:52:58.928896 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.928278    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"
Sep 17 18:52:58.939226 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.939203    2559 scope.go:117] "RemoveContainer" containerID="cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"
Sep 17 18:52:58.939588 ip-10-0-143-176 kubenswrapper[2559]: E0917 18:52:58.939565    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2\": container with ID starting with cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2 not found: ID does not exist" containerID="cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"
Sep 17 18:52:58.939720 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.939600    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2"} err="failed to get container status \"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2\": rpc error: code = NotFound desc = could not find container \"cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2\": container with ID starting with cae8f2018f4cedfbcf41f9169d13d0577f43688a0458693d8587349081d782f2 not found: ID does not exist"
Sep 17 18:52:58.951626 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.951594    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:58.955561 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:58.955535    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-31957-5d489cff66-4lgrj"]
Sep 17 18:52:59.914561 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:59.914530    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 18:52:59.940844 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:52:59.940797    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3134244-4972-426b-9711-39f265dc6585" path="/var/lib/kubelet/pods/a3134244-4972-426b-9711-39f265dc6585/volumes"
Sep 17 18:53:07.814355 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:07.814312    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:53:17.815594 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:17.815513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 18:53:28.602534 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.602497    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 18:53:28.603122 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.603000    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957"
Sep 17 18:53:28.603122 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.603022    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957"
Sep 17 18:53:28.603250 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.603128    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3134244-4972-426b-9711-39f265dc6585" containerName="sequence-graph-31957"
Sep 17 18:53:28.606184 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.606162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.608994 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.608972    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c4e90-serving-cert\""
Sep 17 18:53:28.609104 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.608972    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c4e90-kube-rbac-proxy-sar-config\""
Sep 17 18:53:28.613391 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.613372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:28.617168 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.617148    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 18:53:28.731120 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.731086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.731290 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.731129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.818273 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.818243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:28.832407 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.832380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.832503 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.832417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.833155 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.833123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.834855 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.834820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls\") pod \"sequence-graph-c4e90-775bbd874f-t68tg\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") " pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:28.917001 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:28.916911    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:29.189075 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:29.188989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 18:53:29.192335 ip-10-0-143-176 kubenswrapper[2559]: W0917 18:53:29.192306    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7815cc_b0ef_482e_b011_3effa1fa423d.slice/crio-dcdcfc40a6d0de62ee93b2bdfa1e6f3a2919ae0a08662662f782b6809c75f382 WatchSource:0}: Error finding container dcdcfc40a6d0de62ee93b2bdfa1e6f3a2919ae0a08662662f782b6809c75f382: Status 404 returned error can't find the container with id dcdcfc40a6d0de62ee93b2bdfa1e6f3a2919ae0a08662662f782b6809c75f382
Sep 17 18:53:30.061121 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:30.061087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" event={"ID":"1b7815cc-b0ef-482e-b011-3effa1fa423d","Type":"ContainerStarted","Data":"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"}
Sep 17 18:53:30.061509 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:30.061149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" event={"ID":"1b7815cc-b0ef-482e-b011-3effa1fa423d","Type":"ContainerStarted","Data":"dcdcfc40a6d0de62ee93b2bdfa1e6f3a2919ae0a08662662f782b6809c75f382"}
Sep 17 18:53:30.061509 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:30.061175    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:53:30.082955 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:30.082907    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podStartSLOduration=2.082891184 podStartE2EDuration="2.082891184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:30.081743435 +0000 UTC m=+1452.631967321" watchObservedRunningTime="2026-09-17 18:53:30.082891184 +0000 UTC m=+1452.633115070"
Sep 17 18:53:36.070374 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:53:36.070345    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 18:54:17.927268 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:54:17.927244    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:54:17.931291 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:54:17.931271    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:54:36.839129 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:54:36.839070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:17.954064 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:59:17.954035    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 18:59:17.958801 ip-10-0-143-176 kubenswrapper[2559]: I0917 18:59:17.958775    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:01:06.937692 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:06.937601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 19:01:06.938214 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:06.938071    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" containerID="cri-o://87d471c3ee3c32292498426ede6e8ecbcf0e015403c78308ed74b308d7c00f85" gracePeriod=30
Sep 17 19:01:06.991018 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:06.990987    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:07.090461 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.090424    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 19:01:07.090749 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.090704    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" containerID="cri-o://8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c" gracePeriod=30
Sep 17 19:01:07.090877 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.090760    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kube-rbac-proxy" containerID="cri-o://2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52" gracePeriod=30
Sep 17 19:01:07.306600 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.306564    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:01:07.310103 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.310085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.312487 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.312461    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77f5f-kube-rbac-proxy-sar-config\""
Sep 17 19:01:07.312613 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.312530    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77f5f-predictor-serving-cert\""
Sep 17 19:01:07.318776 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.318752    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:01:07.402800 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.402765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.403034 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.402821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.403034 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.402927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m94\" (UniqueName: \"kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.504115 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.504081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.504296 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.504129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.504296 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.504195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m94\" (UniqueName: \"kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.504387 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:07.504369    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-serving-cert: secret "error-404-isvc-77f5f-predictor-serving-cert" not found
Sep 17 19:01:07.504481 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:07.504469    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls podName:9007de55-362d-4211-b9a0-080799424347 nodeName:}" failed. No retries permitted until 2026-09-17 19:01:08.004446722 +0000 UTC m=+1910.554670601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls") pod "error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" (UID: "9007de55-362d-4211-b9a0-080799424347") : secret "error-404-isvc-77f5f-predictor-serving-cert" not found
Sep 17 19:01:07.504861 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.504825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.515974 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.515950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m94\" (UniqueName: \"kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:07.924027 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.923934    2559 generic.go:356] "Generic (PLEG): container finished" podID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerID="2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52" exitCode=2
Sep 17 19:01:07.924027 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:07.924001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerDied","Data":"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"}
Sep 17 19:01:08.009198 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.009153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:08.011691 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.011665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") pod \"error-404-isvc-77f5f-predictor-6995f557bc-s5ttt\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") " pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:08.220298 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.220202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:08.244620 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.244567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:08.496401 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.496229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:01:08.499032 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:01:08.498997    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9007de55_362d_4211_b9a0_080799424347.slice/crio-98a3925af778817e1f551f700f51bbb9658717430bb4349912b5f306ae0efcd4 WatchSource:0}: Error finding container 98a3925af778817e1f551f700f51bbb9658717430bb4349912b5f306ae0efcd4: Status 404 returned error can't find the container with id 98a3925af778817e1f551f700f51bbb9658717430bb4349912b5f306ae0efcd4
Sep 17 19:01:08.930252 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.930216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerStarted","Data":"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"}
Sep 17 19:01:08.930467 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.930274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerStarted","Data":"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"}
Sep 17 19:01:08.930467 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.930294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerStarted","Data":"98a3925af778817e1f551f700f51bbb9658717430bb4349912b5f306ae0efcd4"}
Sep 17 19:01:08.930467 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.930382    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:08.951121 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:08.951069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podStartSLOduration=1.951051731 podStartE2EDuration="1.951051731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:08.949697676 +0000 UTC m=+1911.499921574" watchObservedRunningTime="2026-09-17 19:01:08.951051731 +0000 UTC m=+1911.501275675"
Sep 17 19:01:09.637053 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:09.637008    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 19:01:09.642334 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:09.642309    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 19:01:09.912797 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:09.912709    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:09.934750 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:09.934719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:09.935910 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:09.935884    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:10.344868 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.344841    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 19:01:10.529705 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.529661    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config\") pod \"43e8da0e-1bf4-434c-b658-df2c140dffc2\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") "
Sep 17 19:01:10.529929 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.529810    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls\") pod \"43e8da0e-1bf4-434c-b658-df2c140dffc2\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") "
Sep 17 19:01:10.529929 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.529870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65l8x\" (UniqueName: \"kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x\") pod \"43e8da0e-1bf4-434c-b658-df2c140dffc2\" (UID: \"43e8da0e-1bf4-434c-b658-df2c140dffc2\") "
Sep 17 19:01:10.530125 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.530102    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43e8da0e-1bf4-434c-b658-df2c140dffc2" (UID: "43e8da0e-1bf4-434c-b658-df2c140dffc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:10.531954 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.531924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43e8da0e-1bf4-434c-b658-df2c140dffc2" (UID: "43e8da0e-1bf4-434c-b658-df2c140dffc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:10.532049 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.531964    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x" (OuterVolumeSpecName: "kube-api-access-65l8x") pod "43e8da0e-1bf4-434c-b658-df2c140dffc2" (UID: "43e8da0e-1bf4-434c-b658-df2c140dffc2"). InnerVolumeSpecName "kube-api-access-65l8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:10.631198 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.631166    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e8da0e-1bf4-434c-b658-df2c140dffc2-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.631198 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.631194    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l8x\" (UniqueName: \"kubernetes.io/projected/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-api-access-65l8x\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.631198 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.631206    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e8da0e-1bf4-434c-b658-df2c140dffc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.939933 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.939820    2559 generic.go:356] "Generic (PLEG): container finished" podID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerID="8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c" exitCode=0
Sep 17 19:01:10.939933 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.939876    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"
Sep 17 19:01:10.939933 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.939901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerDied","Data":"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"}
Sep 17 19:01:10.940519 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.939960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr" event={"ID":"43e8da0e-1bf4-434c-b658-df2c140dffc2","Type":"ContainerDied","Data":"126d3b0620b05da7e72e9930b50ab80a983cbe70cf98979ca8e567948dc104cc"}
Sep 17 19:01:10.940519 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.939990    2559 scope.go:117] "RemoveContainer" containerID="2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"
Sep 17 19:01:10.940519 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.940505    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:10.949290 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.949265    2559 scope.go:117] "RemoveContainer" containerID="8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"
Sep 17 19:01:10.957015 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.956996    2559 scope.go:117] "RemoveContainer" containerID="2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"
Sep 17 19:01:10.957284 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:10.957266    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52\": container with ID starting with 2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52 not found: ID does not exist" containerID="2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"
Sep 17 19:01:10.957364 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.957294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52"} err="failed to get container status \"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52\": rpc error: code = NotFound desc = could not find container \"2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52\": container with ID starting with 2258f1df09aba34dd84c24eab4560dcc792fcd7d4c9b4106aa8ec5d4c0438b52 not found: ID does not exist"
Sep 17 19:01:10.957364 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.957312    2559 scope.go:117] "RemoveContainer" containerID="8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"
Sep 17 19:01:10.957562 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:10.957545    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c\": container with ID starting with 8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c not found: ID does not exist" containerID="8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"
Sep 17 19:01:10.957598 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.957569    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c"} err="failed to get container status \"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c\": rpc error: code = NotFound desc = could not find container \"8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c\": container with ID starting with 8aedbc9ee8bd2983e6fcf46642f3aa6aa04410faf425dffed176e6b390a03a9c not found: ID does not exist"
Sep 17 19:01:10.961962 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.961943    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 19:01:10.966008 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:10.965988    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10598-predictor-577cdb5f5f-xf8dr"]
Sep 17 19:01:11.945927 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:11.945893    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" path="/var/lib/kubelet/pods/43e8da0e-1bf4-434c-b658-df2c140dffc2/volumes"
Sep 17 19:01:14.912713 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:14.912677    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:15.944757 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:15.944729    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:15.945460 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:15.945428    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:19.913785 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:19.913741    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:19.914261 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:19.913899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 19:01:24.913498 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:24.913450    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:25.946299 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:25.946257    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:29.913114 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:29.913074    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:34.913110 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:34.913073    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:35.946047 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:35.946012    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:37.055942 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.055910    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerID="87d471c3ee3c32292498426ede6e8ecbcf0e015403c78308ed74b308d7c00f85" exitCode=0
Sep 17 19:01:37.056408 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.055982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" event={"ID":"1e021e88-3fdb-4afb-b7ed-88bb95340dfa","Type":"ContainerDied","Data":"87d471c3ee3c32292498426ede6e8ecbcf0e015403c78308ed74b308d7c00f85"}
Sep 17 19:01:37.079659 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.079633    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 19:01:37.163823 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.163788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls\") pod \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") "
Sep 17 19:01:37.164007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.163889    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle\") pod \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\" (UID: \"1e021e88-3fdb-4afb-b7ed-88bb95340dfa\") "
Sep 17 19:01:37.164261 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.164229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1e021e88-3fdb-4afb-b7ed-88bb95340dfa" (UID: "1e021e88-3fdb-4afb-b7ed-88bb95340dfa"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:37.165839 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.165802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e021e88-3fdb-4afb-b7ed-88bb95340dfa" (UID: "1e021e88-3fdb-4afb-b7ed-88bb95340dfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:37.265350 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.265314    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:37.265350 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:37.265348    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e021e88-3fdb-4afb-b7ed-88bb95340dfa-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:38.067163 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:38.067123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp" event={"ID":"1e021e88-3fdb-4afb-b7ed-88bb95340dfa","Type":"ContainerDied","Data":"a443176210aab6f4aa126592a97156aab13e8d50f258153e73103d7b2c51d239"}
Sep 17 19:01:38.067163 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:38.067149    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"
Sep 17 19:01:38.067726 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:38.067224    2559 scope.go:117] "RemoveContainer" containerID="87d471c3ee3c32292498426ede6e8ecbcf0e015403c78308ed74b308d7c00f85"
Sep 17 19:01:38.083578 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:38.083553    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 19:01:38.088672 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:38.088648    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-10598-647cf68fb6-9lwpp"]
Sep 17 19:01:39.941405 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:39.941370    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" path="/var/lib/kubelet/pods/1e021e88-3fdb-4afb-b7ed-88bb95340dfa/volumes"
Sep 17 19:01:42.995008 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:42.994972    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 19:01:42.995426 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:42.995240    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" containerID="cri-o://5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60" gracePeriod=30
Sep 17 19:01:43.016089 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.016060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.130710 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.130676    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 19:01:43.131011 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.130964    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container" containerID="cri-o://c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914" gracePeriod=30
Sep 17 19:01:43.131094 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.131001    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kube-rbac-proxy" containerID="cri-o://1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148" gracePeriod=30
Sep 17 19:01:43.140019 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.139996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.228495 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.228459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.307540 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.307507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.342661 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.342626    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:01:43.343012 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.342999    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598"
Sep 17 19:01:43.343066 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343013    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598"
Sep 17 19:01:43.343066 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container"
Sep 17 19:01:43.343066 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343032    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container"
Sep 17 19:01:43.343066 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343043    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kube-rbac-proxy"
Sep 17 19:01:43.343066 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343052    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kube-rbac-proxy"
Sep 17 19:01:43.343224 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343113    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kube-rbac-proxy"
Sep 17 19:01:43.343224 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343123    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e8da0e-1bf4-434c-b658-df2c140dffc2" containerName="kserve-container"
Sep 17 19:01:43.343224 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.343132    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e021e88-3fdb-4afb-b7ed-88bb95340dfa" containerName="ensemble-graph-10598"
Sep 17 19:01:43.346145 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.346131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.348527 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.348497    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-29aad-kube-rbac-proxy-sar-config\""
Sep 17 19:01:43.348683 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.348536    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-29aad-predictor-serving-cert\""
Sep 17 19:01:43.357093 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.357059    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:01:43.385668 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.385623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.418271 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.418240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.418441 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.418294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5ff\" (UniqueName: \"kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.418441 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.418401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.448330 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.448297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:43.519379 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.519345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.519559 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.519394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5ff\" (UniqueName: \"kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.519559 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.519433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.520064 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.520040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.522023 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.521998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.527195 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.527176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5ff\" (UniqueName: \"kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff\") pod \"error-404-isvc-29aad-predictor-658758c67f-4c92z\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") " pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.657554 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.657517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:43.929472 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:43.929389    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:01:43.932321 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:01:43.932290    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ced5fb_aad4_4cb5_a6fa_d2c8800e62b5.slice/crio-e1c556cde61e34b6b3e24aff8b488c13600b324b0df93ad7c63470f23e4a002d WatchSource:0}: Error finding container e1c556cde61e34b6b3e24aff8b488c13600b324b0df93ad7c63470f23e4a002d: Status 404 returned error can't find the container with id e1c556cde61e34b6b3e24aff8b488c13600b324b0df93ad7c63470f23e4a002d
Sep 17 19:01:44.095515 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.095469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerStarted","Data":"18711313d30f5e14d3a22bc69f56b40ebf32ae8abebdb204544a66eaee891e15"}
Sep 17 19:01:44.096007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.095547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerStarted","Data":"14b9bb8a4d23ca88e5b504b8d0f88e28b4786cf7e7af022c87bdd6b4be66576c"}
Sep 17 19:01:44.096007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.095572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerStarted","Data":"e1c556cde61e34b6b3e24aff8b488c13600b324b0df93ad7c63470f23e4a002d"}
Sep 17 19:01:44.096007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.095595    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:44.096007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.095621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:44.097404 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.097370    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:01:44.097511 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.097431    2559 generic.go:356] "Generic (PLEG): container finished" podID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerID="1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148" exitCode=2
Sep 17 19:01:44.097511 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.097500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerDied","Data":"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"}
Sep 17 19:01:44.117254 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:44.117202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podStartSLOduration=1.117183553 podStartE2EDuration="1.117183553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:44.116444759 +0000 UTC m=+1946.666668636" watchObservedRunningTime="2026-09-17 19:01:44.117183553 +0000 UTC m=+1946.667407440"
Sep 17 19:01:45.102629 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:45.102589    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:01:45.945547 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:45.945512    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:01:46.068701 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.068658    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:46.378146 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.378122    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 19:01:46.547020 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.546989    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhqs\" (UniqueName: \"kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs\") pod \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") "
Sep 17 19:01:46.547240 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.547038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config\") pod \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") "
Sep 17 19:01:46.547240 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.547065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") pod \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\" (UID: \"99a9d39b-0547-41c1-93a7-42ce6a71ea15\") "
Sep 17 19:01:46.547470 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.547441    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99a9d39b-0547-41c1-93a7-42ce6a71ea15" (UID: "99a9d39b-0547-41c1-93a7-42ce6a71ea15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:46.549237 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.549205    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs" (OuterVolumeSpecName: "kube-api-access-bkhqs") pod "99a9d39b-0547-41c1-93a7-42ce6a71ea15" (UID: "99a9d39b-0547-41c1-93a7-42ce6a71ea15"). InnerVolumeSpecName "kube-api-access-bkhqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:46.549356 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.549247    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99a9d39b-0547-41c1-93a7-42ce6a71ea15" (UID: "99a9d39b-0547-41c1-93a7-42ce6a71ea15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:46.648174 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.648131    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhqs\" (UniqueName: \"kubernetes.io/projected/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-api-access-bkhqs\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:46.648174 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.648161    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99a9d39b-0547-41c1-93a7-42ce6a71ea15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:46.648174 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:46.648172    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99a9d39b-0547-41c1-93a7-42ce6a71ea15-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:01:47.112844 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.112804    2559 generic.go:356] "Generic (PLEG): container finished" podID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerID="c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914" exitCode=0
Sep 17 19:01:47.113019 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.112859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerDied","Data":"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"}
Sep 17 19:01:47.113019 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.112870    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"
Sep 17 19:01:47.113019 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.112918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr" event={"ID":"99a9d39b-0547-41c1-93a7-42ce6a71ea15","Type":"ContainerDied","Data":"635f1c456d4f635eda9a20d70c1fa0d011376ea792502433d80bbb776bd6265d"}
Sep 17 19:01:47.113019 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.112948    2559 scope.go:117] "RemoveContainer" containerID="1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"
Sep 17 19:01:47.121196 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.121181    2559 scope.go:117] "RemoveContainer" containerID="c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"
Sep 17 19:01:47.128537 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.128521    2559 scope.go:117] "RemoveContainer" containerID="1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"
Sep 17 19:01:47.128788 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:47.128764    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148\": container with ID starting with 1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148 not found: ID does not exist" containerID="1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"
Sep 17 19:01:47.128868 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.128798    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148"} err="failed to get container status \"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148\": rpc error: code = NotFound desc = could not find container \"1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148\": container with ID starting with 1d8561fa0c22a8c1fcf18859525dbb932b8a5bfd962d3293884e26c05192c148 not found: ID does not exist"
Sep 17 19:01:47.128868 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.128814    2559 scope.go:117] "RemoveContainer" containerID="c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"
Sep 17 19:01:47.129053 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:01:47.129036    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914\": container with ID starting with c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914 not found: ID does not exist" containerID="c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"
Sep 17 19:01:47.129090 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.129061    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914"} err="failed to get container status \"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914\": rpc error: code = NotFound desc = could not find container \"c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914\": container with ID starting with c49c3d9527f24ae08ecafe0afc6a3ba0536a142060bc30ccb4eeb084dae48914 not found: ID does not exist"
Sep 17 19:01:47.138797 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.138770    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 19:01:47.144515 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.144495    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4e90-predictor-555f9566bc-fnlzr"]
Sep 17 19:01:47.940678 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:47.940645    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" path="/var/lib/kubelet/pods/99a9d39b-0547-41c1-93a7-42ce6a71ea15/volumes"
Sep 17 19:01:50.107025 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:50.106996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:01:50.107569 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:50.107544    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:01:51.068930 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:51.068890    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:55.945972 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:55.945944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:01:56.069447 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:56.069410    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:01:56.069611 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:01:56.069546    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 19:02:00.108097 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:00.108058    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:02:01.069285 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:01.069251    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:06.069327 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:06.069288    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:07.197404 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.197365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:07.197904 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.197877    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container"
Sep 17 19:02:07.197904 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.197897    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container"
Sep 17 19:02:07.198032 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.197911    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kube-rbac-proxy"
Sep 17 19:02:07.198032 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.197920    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kube-rbac-proxy"
Sep 17 19:02:07.198032 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.198004    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kube-rbac-proxy"
Sep 17 19:02:07.198032 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.198020    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a9d39b-0547-41c1-93a7-42ce6a71ea15" containerName="kserve-container"
Sep 17 19:02:07.200463 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.200442    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.202949 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.202924    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-77f5f-serving-cert\""
Sep 17 19:02:07.203513 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.203496    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-77f5f-kube-rbac-proxy-sar-config\""
Sep 17 19:02:07.205016 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.204988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.205128 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.205042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.211481 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.211458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:07.211755 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.211717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:07.305537 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.305503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.305720 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.305547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.305720 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:02:07.305673    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-77f5f-serving-cert: secret "splitter-graph-77f5f-serving-cert" not found
Sep 17 19:02:07.305858 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:02:07.305757    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls podName:c8521bca-7680-47ea-b135-8a8e2a918b09 nodeName:}" failed. No retries permitted until 2026-09-17 19:02:07.805734782 +0000 UTC m=+1970.355958646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls") pod "splitter-graph-77f5f-d7ff746f7-hwnc6" (UID: "c8521bca-7680-47ea-b135-8a8e2a918b09") : secret "splitter-graph-77f5f-serving-cert" not found
Sep 17 19:02:07.306154 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.306136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.795108 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.795075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:07.809172 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.809142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:07.811547 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:07.811519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") pod \"splitter-graph-77f5f-d7ff746f7-hwnc6\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") " pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:08.111082 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:08.110985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:08.395153 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:08.395071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:08.399250 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:02:08.399216    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8521bca_7680_47ea_b135_8a8e2a918b09.slice/crio-17cbd13289fdbf20a5400ce67d17e43c9f8a67b5045157794f0724df5b3d95d2 WatchSource:0}: Error finding container 17cbd13289fdbf20a5400ce67d17e43c9f8a67b5045157794f0724df5b3d95d2: Status 404 returned error can't find the container with id 17cbd13289fdbf20a5400ce67d17e43c9f8a67b5045157794f0724df5b3d95d2
Sep 17 19:02:09.205960 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:09.205922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" event={"ID":"c8521bca-7680-47ea-b135-8a8e2a918b09","Type":"ContainerStarted","Data":"91620c0b343864cbe08fe280c26adf270f0d821a6ff9dbd90397c7f8f1e92f5a"}
Sep 17 19:02:09.206220 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:09.205970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" event={"ID":"c8521bca-7680-47ea-b135-8a8e2a918b09","Type":"ContainerStarted","Data":"17cbd13289fdbf20a5400ce67d17e43c9f8a67b5045157794f0724df5b3d95d2"}
Sep 17 19:02:09.206220 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:09.206064    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:09.223493 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:09.223439    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podStartSLOduration=2.223426036 podStartE2EDuration="2.223426036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:09.22245871 +0000 UTC m=+1971.772682601" watchObservedRunningTime="2026-09-17 19:02:09.223426036 +0000 UTC m=+1971.773649924"
Sep 17 19:02:10.107794 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:10.107751    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:02:11.068693 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:11.068652    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:13.141687 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.141662    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 19:02:13.149277 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.149254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls\") pod \"1b7815cc-b0ef-482e-b011-3effa1fa423d\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") "
Sep 17 19:02:13.149432 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.149347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle\") pod \"1b7815cc-b0ef-482e-b011-3effa1fa423d\" (UID: \"1b7815cc-b0ef-482e-b011-3effa1fa423d\") "
Sep 17 19:02:13.149713 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.149688    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1b7815cc-b0ef-482e-b011-3effa1fa423d" (UID: "1b7815cc-b0ef-482e-b011-3effa1fa423d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:13.151322 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.151300    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b7815cc-b0ef-482e-b011-3effa1fa423d" (UID: "1b7815cc-b0ef-482e-b011-3effa1fa423d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:13.225040 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.225003    2559 generic.go:356] "Generic (PLEG): container finished" podID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerID="5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60" exitCode=0
Sep 17 19:02:13.225213 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.225050    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"
Sep 17 19:02:13.225213 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.225047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" event={"ID":"1b7815cc-b0ef-482e-b011-3effa1fa423d","Type":"ContainerDied","Data":"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"}
Sep 17 19:02:13.225213 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.225166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg" event={"ID":"1b7815cc-b0ef-482e-b011-3effa1fa423d","Type":"ContainerDied","Data":"dcdcfc40a6d0de62ee93b2bdfa1e6f3a2919ae0a08662662f782b6809c75f382"}
Sep 17 19:02:13.225213 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.225189    2559 scope.go:117] "RemoveContainer" containerID="5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"
Sep 17 19:02:13.233751 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.233734    2559 scope.go:117] "RemoveContainer" containerID="5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"
Sep 17 19:02:13.234129 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:02:13.234109    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60\": container with ID starting with 5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60 not found: ID does not exist" containerID="5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"
Sep 17 19:02:13.234202 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.234139    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60"} err="failed to get container status \"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60\": rpc error: code = NotFound desc = could not find container \"5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60\": container with ID starting with 5866b3245ab6aef044ba69399a0e34dbfcc2013763618c8e96e6ac2637a37e60 not found: ID does not exist"
Sep 17 19:02:13.245207 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.245183    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 19:02:13.249908 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.249848    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7815cc-b0ef-482e-b011-3effa1fa423d-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:13.249908 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.249872    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7815cc-b0ef-482e-b011-3effa1fa423d-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:13.251570 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.251547    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c4e90-775bbd874f-t68tg"]
Sep 17 19:02:13.940175 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:13.940143    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" path="/var/lib/kubelet/pods/1b7815cc-b0ef-482e-b011-3effa1fa423d/volumes"
Sep 17 19:02:15.216134 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:15.216109    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:17.316131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.316098    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:17.316530 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.316282    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" containerID="cri-o://91620c0b343864cbe08fe280c26adf270f0d821a6ff9dbd90397c7f8f1e92f5a" gracePeriod=30
Sep 17 19:02:17.350256 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.350222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.463338 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.463280    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.474528 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.474493    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:02:17.474735 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.474710    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container" containerID="cri-o://7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25" gracePeriod=30
Sep 17 19:02:17.474859 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.474795    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kube-rbac-proxy" containerID="cri-o://35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb" gracePeriod=30
Sep 17 19:02:17.545809 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.545773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.645955 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.645924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.668654 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.668615    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:02:17.668959 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.668947    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90"
Sep 17 19:02:17.669020 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.668960    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90"
Sep 17 19:02:17.669054 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.669026    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7815cc-b0ef-482e-b011-3effa1fa423d" containerName="sequence-graph-c4e90"
Sep 17 19:02:17.672054 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.672037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.674537 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.674514    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5a2c4-predictor-serving-cert\""
Sep 17 19:02:17.674658 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.674540    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5a2c4-kube-rbac-proxy-sar-config\""
Sep 17 19:02:17.681880 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.681857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:02:17.689813 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.689780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.689964 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.689823    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pbt\" (UniqueName: \"kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.689964 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.689948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.748971 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.748915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.791100 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.791062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.791297 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.791138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.791297 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.791163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pbt\" (UniqueName: \"kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.791734 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.791712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.793708 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.793682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.802923 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.802853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pbt\" (UniqueName: \"kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt\") pod \"error-404-isvc-5a2c4-predictor-7d688b9859-9flbs\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") " pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:17.846683 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.846602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.982176 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:17.982140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:18.049971 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:18.049934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:18.250753 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:18.250721    2559 generic.go:356] "Generic (PLEG): container finished" podID="9007de55-362d-4211-b9a0-080799424347" containerID="35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb" exitCode=2
Sep 17 19:02:18.250926 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:18.250762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerDied","Data":"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"}
Sep 17 19:02:18.254743 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:18.254717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:02:18.256498 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:02:18.256467    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb075c7d2_2db1_4d41_a94c_45dbb8f823a9.slice/crio-674ce90c3d6b21dcbeda62002751a244b072c989f3a440e755be8a27407e540e WatchSource:0}: Error finding container 674ce90c3d6b21dcbeda62002751a244b072c989f3a440e755be8a27407e540e: Status 404 returned error can't find the container with id 674ce90c3d6b21dcbeda62002751a244b072c989f3a440e755be8a27407e540e
Sep 17 19:02:19.258483 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:19.258440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerStarted","Data":"7cf1548b29f33dc175a1466a399dde97987f75c1ad322a5e030f28792f609777"}
Sep 17 19:02:19.258906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:19.258509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerStarted","Data":"99df80937db694c23fb006527f928a05cf030eaa5b05d85d5380f5603a2c4005"}
Sep 17 19:02:19.258906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:19.258532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerStarted","Data":"674ce90c3d6b21dcbeda62002751a244b072c989f3a440e755be8a27407e540e"}
Sep 17 19:02:19.258906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:19.258562    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:19.277589 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:19.277540    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podStartSLOduration=2.277525111 podStartE2EDuration="2.277525111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:19.275848095 +0000 UTC m=+1981.826071973" watchObservedRunningTime="2026-09-17 19:02:19.277525111 +0000 UTC m=+1981.827748998"
Sep 17 19:02:20.108176 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.108134    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 19:02:20.214474 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.214433    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:20.262554 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.262521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:20.264056 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.264026    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:02:20.727193 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.727164    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:02:20.819719 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.819632    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config\") pod \"9007de55-362d-4211-b9a0-080799424347\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") "
Sep 17 19:02:20.819719 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.819682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") pod \"9007de55-362d-4211-b9a0-080799424347\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") "
Sep 17 19:02:20.819719 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.819709    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7m94\" (UniqueName: \"kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94\") pod \"9007de55-362d-4211-b9a0-080799424347\" (UID: \"9007de55-362d-4211-b9a0-080799424347\") "
Sep 17 19:02:20.820155 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.820129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9007de55-362d-4211-b9a0-080799424347" (UID: "9007de55-362d-4211-b9a0-080799424347"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:20.821912 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.821888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94" (OuterVolumeSpecName: "kube-api-access-k7m94") pod "9007de55-362d-4211-b9a0-080799424347" (UID: "9007de55-362d-4211-b9a0-080799424347"). InnerVolumeSpecName "kube-api-access-k7m94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:20.821912 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.821905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9007de55-362d-4211-b9a0-080799424347" (UID: "9007de55-362d-4211-b9a0-080799424347"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:20.921131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.921094    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9007de55-362d-4211-b9a0-080799424347-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:20.921131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.921124    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m94\" (UniqueName: \"kubernetes.io/projected/9007de55-362d-4211-b9a0-080799424347-kube-api-access-k7m94\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:20.921131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:20.921133    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9007de55-362d-4211-b9a0-080799424347-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:21.268290 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268254    2559 generic.go:356] "Generic (PLEG): container finished" podID="9007de55-362d-4211-b9a0-080799424347" containerID="7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25" exitCode=0
Sep 17 19:02:21.268725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268306    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"
Sep 17 19:02:21.268725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268335    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerDied","Data":"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"}
Sep 17 19:02:21.268725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt" event={"ID":"9007de55-362d-4211-b9a0-080799424347","Type":"ContainerDied","Data":"98a3925af778817e1f551f700f51bbb9658717430bb4349912b5f306ae0efcd4"}
Sep 17 19:02:21.268725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268423    2559 scope.go:117] "RemoveContainer" containerID="35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"
Sep 17 19:02:21.268969 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.268911    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:02:21.278067 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.278047    2559 scope.go:117] "RemoveContainer" containerID="7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"
Sep 17 19:02:21.285851 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.285821    2559 scope.go:117] "RemoveContainer" containerID="35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"
Sep 17 19:02:21.286138 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:02:21.286120    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb\": container with ID starting with 35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb not found: ID does not exist" containerID="35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"
Sep 17 19:02:21.286196 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.286147    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb"} err="failed to get container status \"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb\": rpc error: code = NotFound desc = could not find container \"35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb\": container with ID starting with 35b5890d09371a1eb98402d5b1a4dfcf2abac651b1a2f4f4688f6f2a51ba75eb not found: ID does not exist"
Sep 17 19:02:21.286196 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.286164    2559 scope.go:117] "RemoveContainer" containerID="7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"
Sep 17 19:02:21.286413 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:02:21.286391    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25\": container with ID starting with 7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25 not found: ID does not exist" containerID="7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"
Sep 17 19:02:21.286480 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.286422    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25"} err="failed to get container status \"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25\": rpc error: code = NotFound desc = could not find container \"7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25\": container with ID starting with 7b9c33efc89bd90656f291b96208d5e3f3d390f69074d4bb170c6f349fad0e25 not found: ID does not exist"
Sep 17 19:02:21.294058 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.294033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:02:21.297699 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.297676    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f5f-predictor-6995f557bc-s5ttt"]
Sep 17 19:02:21.940915 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:21.940886    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9007de55-362d-4211-b9a0-080799424347" path="/var/lib/kubelet/pods/9007de55-362d-4211-b9a0-080799424347/volumes"
Sep 17 19:02:25.214668 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:25.214628    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:26.273674 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:26.273644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:02:26.274105 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:26.274005    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:02:30.109015 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:30.108988    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:02:30.214818 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:30.214782    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:30.215025 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:30.214911    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:35.214781 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:35.214740    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:36.275092 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:36.275051    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:02:40.214853 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:40.214799    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:43.259732 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.259694    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260035    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container"
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260046    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container"
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260068    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kube-rbac-proxy"
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260073    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kube-rbac-proxy"
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260116    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kube-rbac-proxy"
Sep 17 19:02:43.260131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.260125    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9007de55-362d-4211-b9a0-080799424347" containerName="kserve-container"
Sep 17 19:02:43.263346 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.263327    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.264531 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.264480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:43.265969 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.265952    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-29aad-serving-cert\""
Sep 17 19:02:43.266056 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.265952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-29aad-kube-rbac-proxy-sar-config\""
Sep 17 19:02:43.272919 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.272891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:02:43.411173 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.411137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.411367 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.411204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.503245 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.503211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:43.512454 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.512382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.512454 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.512446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.513125 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.513098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.514773 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.514757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls\") pod \"switch-graph-29aad-74c87f6b5c-t62mf\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") " pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.573597 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.573553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:43.846988 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:43.846912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:02:43.851004 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:02:43.850972    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cb3bb1_cb62_439a_98d2_10f2e02ec99a.slice/crio-5f60e5be4a8102aebe8becff8ea828b869ea1d851915ecfd2ac26758471e015a WatchSource:0}: Error finding container 5f60e5be4a8102aebe8becff8ea828b869ea1d851915ecfd2ac26758471e015a: Status 404 returned error can't find the container with id 5f60e5be4a8102aebe8becff8ea828b869ea1d851915ecfd2ac26758471e015a
Sep 17 19:02:44.369286 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:44.369244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" event={"ID":"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a","Type":"ContainerStarted","Data":"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"}
Sep 17 19:02:44.369711 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:44.369296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" event={"ID":"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a","Type":"ContainerStarted","Data":"5f60e5be4a8102aebe8becff8ea828b869ea1d851915ecfd2ac26758471e015a"}
Sep 17 19:02:44.369711 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:44.369384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:44.388328 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:44.388277    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podStartSLOduration=1.388260928 podStartE2EDuration="1.388260928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:44.386594344 +0000 UTC m=+2006.936818232" watchObservedRunningTime="2026-09-17 19:02:44.388260928 +0000 UTC m=+2006.938484814"
Sep 17 19:02:45.215345 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:45.215310    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:46.274279 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:46.274241    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:02:47.386605 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.386567    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerID="91620c0b343864cbe08fe280c26adf270f0d821a6ff9dbd90397c7f8f1e92f5a" exitCode=0
Sep 17 19:02:47.386961 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.386637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" event={"ID":"c8521bca-7680-47ea-b135-8a8e2a918b09","Type":"ContainerDied","Data":"91620c0b343864cbe08fe280c26adf270f0d821a6ff9dbd90397c7f8f1e92f5a"}
Sep 17 19:02:47.463340 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.463317    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:47.548374 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.548332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") pod \"c8521bca-7680-47ea-b135-8a8e2a918b09\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") "
Sep 17 19:02:47.548566 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.548475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle\") pod \"c8521bca-7680-47ea-b135-8a8e2a918b09\" (UID: \"c8521bca-7680-47ea-b135-8a8e2a918b09\") "
Sep 17 19:02:47.548808 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.548780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c8521bca-7680-47ea-b135-8a8e2a918b09" (UID: "c8521bca-7680-47ea-b135-8a8e2a918b09"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:47.550404 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.550371    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8521bca-7680-47ea-b135-8a8e2a918b09" (UID: "c8521bca-7680-47ea-b135-8a8e2a918b09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:47.649614 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.649534    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8521bca-7680-47ea-b135-8a8e2a918b09-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:47.649614 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:47.649562    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8521bca-7680-47ea-b135-8a8e2a918b09-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.392126 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:48.392089    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"
Sep 17 19:02:48.392618 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:48.392088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6" event={"ID":"c8521bca-7680-47ea-b135-8a8e2a918b09","Type":"ContainerDied","Data":"17cbd13289fdbf20a5400ce67d17e43c9f8a67b5045157794f0724df5b3d95d2"}
Sep 17 19:02:48.392618 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:48.392250    2559 scope.go:117] "RemoveContainer" containerID="91620c0b343864cbe08fe280c26adf270f0d821a6ff9dbd90397c7f8f1e92f5a"
Sep 17 19:02:48.434338 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:48.434293    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:48.438927 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:48.438900    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-77f5f-d7ff746f7-hwnc6"]
Sep 17 19:02:49.941158 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:49.941121    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" path="/var/lib/kubelet/pods/c8521bca-7680-47ea-b135-8a8e2a918b09/volumes"
Sep 17 19:02:50.378967 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:50.378936    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:02:56.274276 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:02:56.274235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 19:03:06.275231 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:06.275202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:03:17.601152 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.601108    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:03:17.601705 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.601614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f"
Sep 17 19:03:17.601705 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.601660    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f"
Sep 17 19:03:17.601876 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.601744    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8521bca-7680-47ea-b135-8a8e2a918b09" containerName="splitter-graph-77f5f"
Sep 17 19:03:17.604735 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.604713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.607047 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.607024    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5a2c4-serving-cert\""
Sep 17 19:03:17.607636 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.607618    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5a2c4-kube-rbac-proxy-sar-config\""
Sep 17 19:03:17.617960 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.617938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:03:17.621358 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.621333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:17.685902 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.685867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.686078 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.685930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.718818 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.718789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:17.787152 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.787121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.787325 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.787169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.787747 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.787723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.789497 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.789472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls\") pod \"splitter-graph-5a2c4-77c5594cb8-l58pz\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") " pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:17.914688 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:17.914606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:18.184574 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:18.184509    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:03:18.186765 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:03:18.186723    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834cddad_a17c_4842_a3aa_3c03a6f9471f.slice/crio-5cb661e578c34572c5db6fd382e01452f424730fc7974eeba64bb04bf1cb20c8 WatchSource:0}: Error finding container 5cb661e578c34572c5db6fd382e01452f424730fc7974eeba64bb04bf1cb20c8: Status 404 returned error can't find the container with id 5cb661e578c34572c5db6fd382e01452f424730fc7974eeba64bb04bf1cb20c8
Sep 17 19:03:18.519428 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:18.519395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" event={"ID":"834cddad-a17c-4842-a3aa-3c03a6f9471f","Type":"ContainerStarted","Data":"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"}
Sep 17 19:03:18.519611 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:18.519445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" event={"ID":"834cddad-a17c-4842-a3aa-3c03a6f9471f","Type":"ContainerStarted","Data":"5cb661e578c34572c5db6fd382e01452f424730fc7974eeba64bb04bf1cb20c8"}
Sep 17 19:03:18.519611 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:18.519521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:18.540151 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:18.540102    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podStartSLOduration=1.540088299 podStartE2EDuration="1.540088299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:18.537939332 +0000 UTC m=+2041.088163241" watchObservedRunningTime="2026-09-17 19:03:18.540088299 +0000 UTC m=+2041.090312185"
Sep 17 19:03:24.529001 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:24.528972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:03:39.611256 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:03:39.611225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:17.978725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:04:17.978697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:04:17.986211 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:04:17.986186    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:08:08.202129 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:08:08.202084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:21.360847 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:08:21.360754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:21.426809 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:08:21.426773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:27.419278 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:08:27.419246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:18.004624 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:09:18.004593    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:09:18.012263 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:09:18.012239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:10:12.899765 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:10:12.899727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:32.152562 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.152465    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:11:32.153231 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.152667    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" containerID="cri-o://4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701" gracePeriod=30
Sep 17 19:11:32.174210 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.174170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:32.377116 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.377079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:32.400007 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.399968    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:11:32.404815 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.404677    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kube-rbac-proxy" containerID="cri-o://7cf1548b29f33dc175a1466a399dde97987f75c1ad322a5e030f28792f609777" gracePeriod=30
Sep 17 19:11:32.404815 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.404728    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container" containerID="cri-o://99df80937db694c23fb006527f928a05cf030eaa5b05d85d5380f5603a2c4005" gracePeriod=30
Sep 17 19:11:32.472931 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:32.472893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:33.512143 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:33.512108    2559 generic.go:356] "Generic (PLEG): container finished" podID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerID="7cf1548b29f33dc175a1466a399dde97987f75c1ad322a5e030f28792f609777" exitCode=2
Sep 17 19:11:33.512143 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:33.512139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerDied","Data":"7cf1548b29f33dc175a1466a399dde97987f75c1ad322a5e030f28792f609777"}
Sep 17 19:11:34.527808 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:34.527765    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:11:35.523481 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.523453    2559 generic.go:356] "Generic (PLEG): container finished" podID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerID="99df80937db694c23fb006527f928a05cf030eaa5b05d85d5380f5603a2c4005" exitCode=0
Sep 17 19:11:35.523608 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.523523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerDied","Data":"99df80937db694c23fb006527f928a05cf030eaa5b05d85d5380f5603a2c4005"}
Sep 17 19:11:35.646375 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.646343    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:11:35.737798 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.737700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9pbt\" (UniqueName: \"kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt\") pod \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") "
Sep 17 19:11:35.737798 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.737769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls\") pod \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") "
Sep 17 19:11:35.738076 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.737868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config\") pod \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\" (UID: \"b075c7d2-2db1-4d41-a94c-45dbb8f823a9\") "
Sep 17 19:11:35.738221 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.738190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b075c7d2-2db1-4d41-a94c-45dbb8f823a9" (UID: "b075c7d2-2db1-4d41-a94c-45dbb8f823a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:35.739922 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.739898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b075c7d2-2db1-4d41-a94c-45dbb8f823a9" (UID: "b075c7d2-2db1-4d41-a94c-45dbb8f823a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:35.740033 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.740016    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt" (OuterVolumeSpecName: "kube-api-access-f9pbt") pod "b075c7d2-2db1-4d41-a94c-45dbb8f823a9" (UID: "b075c7d2-2db1-4d41-a94c-45dbb8f823a9"). InnerVolumeSpecName "kube-api-access-f9pbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:35.838985 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.838949    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9pbt\" (UniqueName: \"kubernetes.io/projected/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-api-access-f9pbt\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:11:35.838985 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.838979    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:11:35.838985 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:35.838988    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b075c7d2-2db1-4d41-a94c-45dbb8f823a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:11:36.529373 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.529337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs" event={"ID":"b075c7d2-2db1-4d41-a94c-45dbb8f823a9","Type":"ContainerDied","Data":"674ce90c3d6b21dcbeda62002751a244b072c989f3a440e755be8a27407e540e"}
Sep 17 19:11:36.529373 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.529363    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"
Sep 17 19:11:36.529605 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.529407    2559 scope.go:117] "RemoveContainer" containerID="7cf1548b29f33dc175a1466a399dde97987f75c1ad322a5e030f28792f609777"
Sep 17 19:11:36.537712 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.537689    2559 scope.go:117] "RemoveContainer" containerID="99df80937db694c23fb006527f928a05cf030eaa5b05d85d5380f5603a2c4005"
Sep 17 19:11:36.551074 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.551048    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:11:36.560608 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:36.560581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a2c4-predictor-7d688b9859-9flbs"]
Sep 17 19:11:37.941016 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:37.940981    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" path="/var/lib/kubelet/pods/b075c7d2-2db1-4d41-a94c-45dbb8f823a9/volumes"
Sep 17 19:11:39.528316 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:39.528278    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:11:44.526987 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:44.526946    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:11:44.527372 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:44.527045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:11:49.527554 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:49.527508    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:11:54.528297 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:54.528253    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:11:59.527800 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:11:59.527763    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:12:02.301462 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.301433    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:12:02.351632 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.351595    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls\") pod \"834cddad-a17c-4842-a3aa-3c03a6f9471f\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") "
Sep 17 19:12:02.351803 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.351694    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle\") pod \"834cddad-a17c-4842-a3aa-3c03a6f9471f\" (UID: \"834cddad-a17c-4842-a3aa-3c03a6f9471f\") "
Sep 17 19:12:02.352045 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.352020    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "834cddad-a17c-4842-a3aa-3c03a6f9471f" (UID: "834cddad-a17c-4842-a3aa-3c03a6f9471f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:02.353724 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.353693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "834cddad-a17c-4842-a3aa-3c03a6f9471f" (UID: "834cddad-a17c-4842-a3aa-3c03a6f9471f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:02.453017 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.452926    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834cddad-a17c-4842-a3aa-3c03a6f9471f-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:12:02.453017 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.452961    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834cddad-a17c-4842-a3aa-3c03a6f9471f-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:12:02.641955 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.641916    2559 generic.go:356] "Generic (PLEG): container finished" podID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerID="4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701" exitCode=0
Sep 17 19:12:02.642138 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.641966    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"
Sep 17 19:12:02.642138 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.642003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" event={"ID":"834cddad-a17c-4842-a3aa-3c03a6f9471f","Type":"ContainerDied","Data":"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"}
Sep 17 19:12:02.642138 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.642071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz" event={"ID":"834cddad-a17c-4842-a3aa-3c03a6f9471f","Type":"ContainerDied","Data":"5cb661e578c34572c5db6fd382e01452f424730fc7974eeba64bb04bf1cb20c8"}
Sep 17 19:12:02.642138 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.642096    2559 scope.go:117] "RemoveContainer" containerID="4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"
Sep 17 19:12:02.650601 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.650580    2559 scope.go:117] "RemoveContainer" containerID="4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"
Sep 17 19:12:02.650892 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:12:02.650865    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701\": container with ID starting with 4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701 not found: ID does not exist" containerID="4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"
Sep 17 19:12:02.650985 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.650904    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701"} err="failed to get container status \"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701\": rpc error: code = NotFound desc = could not find container \"4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701\": container with ID starting with 4d3c450ca3d2c5b498a2a9445943699a8c5c6c7467f5855452dd005b75f34701 not found: ID does not exist"
Sep 17 19:12:02.671077 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.671044    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:12:02.677734 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:02.677705    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5a2c4-77c5594cb8-l58pz"]
Sep 17 19:12:03.940419 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:12:03.940387    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" path="/var/lib/kubelet/pods/834cddad-a17c-4842-a3aa-3c03a6f9471f/volumes"
Sep 17 19:13:10.607849 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:13:10.607797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:18.030298 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:14:18.030211    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:14:18.043754 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:14:18.043730    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:19:02.461600 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.461513    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:19:02.464117 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.461744    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" containerID="cri-o://a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb" gracePeriod=30
Sep 17 19:19:02.540811 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.540768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:02.677816 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.677775    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:19:02.678069 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.678019    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container" containerID="cri-o://14b9bb8a4d23ca88e5b504b8d0f88e28b4786cf7e7af022c87bdd6b4be66576c" gracePeriod=30
Sep 17 19:19:02.678069 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:02.678061    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kube-rbac-proxy" containerID="cri-o://18711313d30f5e14d3a22bc69f56b40ebf32ae8abebdb204544a66eaee891e15" gracePeriod=30
Sep 17 19:19:03.282723 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.282691    2559 generic.go:356] "Generic (PLEG): container finished" podID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerID="18711313d30f5e14d3a22bc69f56b40ebf32ae8abebdb204544a66eaee891e15" exitCode=2
Sep 17 19:19:03.282945 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.282758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerDied","Data":"18711313d30f5e14d3a22bc69f56b40ebf32ae8abebdb204544a66eaee891e15"}
Sep 17 19:19:03.478973 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.478941    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cs49d/must-gather-kjd2l"]
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479246    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kube-rbac-proxy"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479257    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kube-rbac-proxy"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479269    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479276    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479293    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479299    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container"
Sep 17 19:19:03.479353 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479352    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kube-rbac-proxy"
Sep 17 19:19:03.479621 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479360    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b075c7d2-2db1-4d41-a94c-45dbb8f823a9" containerName="kserve-container"
Sep 17 19:19:03.479621 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.479367    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="834cddad-a17c-4842-a3aa-3c03a6f9471f" containerName="splitter-graph-5a2c4"
Sep 17 19:19:03.482435 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.482418    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.485336 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.485317    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs49d\"/\"kube-root-ca.crt\""
Sep 17 19:19:03.485458 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.485438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cs49d\"/\"default-dockercfg-fhpsr\""
Sep 17 19:19:03.486314 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.486295    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs49d\"/\"openshift-service-ca.crt\""
Sep 17 19:19:03.493814 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.493788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs49d/must-gather-kjd2l"]
Sep 17 19:19:03.621526 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.621437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbds\" (UniqueName: \"kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.621526 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.621484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.722977 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.722925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbds\" (UniqueName: \"kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.723163 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.722991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.723364 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.723343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.735769 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.735744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbds\" (UniqueName: \"kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds\") pod \"must-gather-kjd2l\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") " pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.804659 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.804626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:03.973534 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:03.973432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:04.088575 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:04.088536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs49d/must-gather-kjd2l"]
Sep 17 19:19:04.091610 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:19:04.091581    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51cd86bc_bcbf_4ad8_9155_8a6249ded021.slice/crio-9913d19cf5f2b5a951412a80a643329fff86e372946d099c620605c5e93a55a9 WatchSource:0}: Error finding container 9913d19cf5f2b5a951412a80a643329fff86e372946d099c620605c5e93a55a9: Status 404 returned error can't find the container with id 9913d19cf5f2b5a951412a80a643329fff86e372946d099c620605c5e93a55a9
Sep 17 19:19:04.093386 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:04.093366    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:19:04.287567 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:04.287530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs49d/must-gather-kjd2l" event={"ID":"51cd86bc-bcbf-4ad8-9155-8a6249ded021","Type":"ContainerStarted","Data":"9913d19cf5f2b5a951412a80a643329fff86e372946d099c620605c5e93a55a9"}
Sep 17 19:19:05.103855 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:05.103523    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 19:19:05.381266 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:05.380946    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:05.993982 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:05.993941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:06.300542 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:06.300505    2559 generic.go:356] "Generic (PLEG): container finished" podID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerID="14b9bb8a4d23ca88e5b504b8d0f88e28b4786cf7e7af022c87bdd6b4be66576c" exitCode=0
Sep 17 19:19:06.300542 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:06.300548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerDied","Data":"14b9bb8a4d23ca88e5b504b8d0f88e28b4786cf7e7af022c87bdd6b4be66576c"}
Sep 17 19:19:09.276370 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.276340    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:19:09.317051 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.317011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z" event={"ID":"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5","Type":"ContainerDied","Data":"e1c556cde61e34b6b3e24aff8b488c13600b324b0df93ad7c63470f23e4a002d"}
Sep 17 19:19:09.317240 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.317094    2559 scope.go:117] "RemoveContainer" containerID="18711313d30f5e14d3a22bc69f56b40ebf32ae8abebdb204544a66eaee891e15"
Sep 17 19:19:09.317240 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.317094    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"
Sep 17 19:19:09.377319 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.377282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq5ff\" (UniqueName: \"kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff\") pod \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") "
Sep 17 19:19:09.377508 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.377341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config\") pod \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") "
Sep 17 19:19:09.377508 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.377388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls\") pod \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\" (UID: \"71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5\") "
Sep 17 19:19:09.377809 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.377780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" (UID: "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:09.379772 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.379747    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff" (OuterVolumeSpecName: "kube-api-access-dq5ff") pod "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" (UID: "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5"). InnerVolumeSpecName "kube-api-access-dq5ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:09.380127 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.380097    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" (UID: "71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:09.478281 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.478240    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:09.478281 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.478273    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5ff\" (UniqueName: \"kubernetes.io/projected/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-api-access-dq5ff\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:09.478281 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.478286    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:09.641638 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.641600    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:19:09.661620 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.661581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-29aad-predictor-658758c67f-4c92z"]
Sep 17 19:19:09.942899 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:09.942803    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" path="/var/lib/kubelet/pods/71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5/volumes"
Sep 17 19:19:10.149534 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:10.149512    2559 scope.go:117] "RemoveContainer" containerID="14b9bb8a4d23ca88e5b504b8d0f88e28b4786cf7e7af022c87bdd6b4be66576c"
Sep 17 19:19:10.377817 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:10.377781    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:11.334516 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:11.334479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs49d/must-gather-kjd2l" event={"ID":"51cd86bc-bcbf-4ad8-9155-8a6249ded021","Type":"ContainerStarted","Data":"ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306"}
Sep 17 19:19:11.334717 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:11.334527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs49d/must-gather-kjd2l" event={"ID":"51cd86bc-bcbf-4ad8-9155-8a6249ded021","Type":"ContainerStarted","Data":"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"}
Sep 17 19:19:11.355729 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:11.355677    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cs49d/must-gather-kjd2l" podStartSLOduration=2.186139158 podStartE2EDuration="8.355661066s" totalImagesPullingTime="6.169521908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:11.355577569 +0000 UTC m=+2993.905801458" watchObservedRunningTime="2026-09-17 19:19:11.355661066 +0000 UTC m=+2993.905884952"
Sep 17 19:19:15.378657 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:15.378621    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:15.379075 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:15.378726    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:19:18.063039 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:18.063006    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:19:18.072865 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:18.072820    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:19:19.268283 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:19.268251    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:20.203240 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:20.203210    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:20.379699 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:20.379653    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:21.006813 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:21.006780    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:21.775847 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:21.775813    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:22.557973 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:22.557936    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:23.415603 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:23.415567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:24.359089 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:24.359053    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:25.131298 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:25.131268    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:25.378310 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:25.378270    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:25.936879 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:25.936822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:26.730466 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:26.730415    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:27.548864 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:27.548791    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:28.487602 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:28.487555    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:29.408725 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:29.408687    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:30.377360 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:30.377323    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:30.487934 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:30.487905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:31.409598 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:31.409567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-29aad-74c87f6b5c-t62mf_f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/switch-graph-29aad/0.log"
Sep 17 19:19:32.433189 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:32.433158    2559 generic.go:356] "Generic (PLEG): container finished" podID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerID="23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039" exitCode=0
Sep 17 19:19:32.433604 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:32.433238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs49d/must-gather-kjd2l" event={"ID":"51cd86bc-bcbf-4ad8-9155-8a6249ded021","Type":"ContainerDied","Data":"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"}
Sep 17 19:19:32.433604 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:32.433580    2559 scope.go:117] "RemoveContainer" containerID="23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"
Sep 17 19:19:33.104271 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.104244    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:19:33.184017 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.183981    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle\") pod \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") "
Sep 17 19:19:33.184192 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.184069    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls\") pod \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\" (UID: \"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a\") "
Sep 17 19:19:33.184429 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.184402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" (UID: "f2cb3bb1-cb62-439a-98d2-10f2e02ec99a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:33.186154 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.186136    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" (UID: "f2cb3bb1-cb62-439a-98d2-10f2e02ec99a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:33.285367 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.285339    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-proxy-tls\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.285367 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.285369    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a-openshift-service-ca-bundle\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.351742 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.351707    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cs49d_must-gather-kjd2l_51cd86bc-bcbf-4ad8-9155-8a6249ded021/gather/0.log"
Sep 17 19:19:33.437846 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.437807    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerID="a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb" exitCode=0
Sep 17 19:19:33.438243 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.437867    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"
Sep 17 19:19:33.438243 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.437864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" event={"ID":"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a","Type":"ContainerDied","Data":"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"}
Sep 17 19:19:33.438243 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.437992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf" event={"ID":"f2cb3bb1-cb62-439a-98d2-10f2e02ec99a","Type":"ContainerDied","Data":"5f60e5be4a8102aebe8becff8ea828b869ea1d851915ecfd2ac26758471e015a"}
Sep 17 19:19:33.438243 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.438015    2559 scope.go:117] "RemoveContainer" containerID="a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"
Sep 17 19:19:33.446398 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.446377    2559 scope.go:117] "RemoveContainer" containerID="a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"
Sep 17 19:19:33.446645 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:33.446628    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb\": container with ID starting with a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb not found: ID does not exist" containerID="a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"
Sep 17 19:19:33.446698 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.446653    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb"} err="failed to get container status \"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb\": rpc error: code = NotFound desc = could not find container \"a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb\": container with ID starting with a0ef396f356194d5dc132178799e11456fa718e3cab2206f139fdddd47ab40fb not found: ID does not exist"
Sep 17 19:19:33.461915 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.461890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:19:33.466340 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.466318    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-29aad-74c87f6b5c-t62mf"]
Sep 17 19:19:33.883454 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.883377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:33.940308 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:33.940266    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" path="/var/lib/kubelet/pods/f2cb3bb1-cb62-439a-98d2-10f2e02ec99a/volumes"
Sep 17 19:19:34.044639 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:34.044600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:34.149235 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:34.149132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:37.198696 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:37.198649    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rf6t6_923b92fd-9415-43ec-ab08-465f6cd44407/global-pull-secret-syncer/0.log"
Sep 17 19:19:37.350284 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:37.350248    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log"
Sep 17 19:19:37.546187 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:37.546152    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-176.ec2.internal_53df0d23cf6bfbf0621c183fbd5dbe70/haproxy/0.log"
Sep 17 19:19:38.854664 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:38.854629    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-cs49d/must-gather-kjd2l"]
Sep 17 19:19:38.855189 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:38.854885    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-cs49d/must-gather-kjd2l" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="copy" containerID="cri-o://ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306" gracePeriod=2
Sep 17 19:19:38.860372 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:38.860347    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-cs49d/must-gather-kjd2l"]
Sep 17 19:19:39.079824 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.079802    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cs49d_must-gather-kjd2l_51cd86bc-bcbf-4ad8-9155-8a6249ded021/copy/0.log"
Sep 17 19:19:39.081459 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.081439    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:39.084092 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:39.084067    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kjd2l\" is forbidden: User \"system:node:ip-10-0-143-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cs49d\": no relationship found between node 'ip-10-0-143-176.ec2.internal' and this object" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:39.129551 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.129461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output\") pod \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") "
Sep 17 19:19:39.129551 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.129547    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbds\" (UniqueName: \"kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds\") pod \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\" (UID: \"51cd86bc-bcbf-4ad8-9155-8a6249ded021\") "
Sep 17 19:19:39.131033 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.130998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "51cd86bc-bcbf-4ad8-9155-8a6249ded021" (UID: "51cd86bc-bcbf-4ad8-9155-8a6249ded021"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:39.131720 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.131704    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds" (OuterVolumeSpecName: "kube-api-access-lkbds") pod "51cd86bc-bcbf-4ad8-9155-8a6249ded021" (UID: "51cd86bc-bcbf-4ad8-9155-8a6249ded021"). InnerVolumeSpecName "kube-api-access-lkbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:39.230140 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.230099    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbds\" (UniqueName: \"kubernetes.io/projected/51cd86bc-bcbf-4ad8-9155-8a6249ded021-kube-api-access-lkbds\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:39.230140 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.230129    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd86bc-bcbf-4ad8-9155-8a6249ded021-must-gather-output\") on node \"ip-10-0-143-176.ec2.internal\" DevicePath \"\""
Sep 17 19:19:39.462790 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.462702    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cs49d_must-gather-kjd2l_51cd86bc-bcbf-4ad8-9155-8a6249ded021/copy/0.log"
Sep 17 19:19:39.463125 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.463099    2559 generic.go:356] "Generic (PLEG): container finished" podID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerID="ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306" exitCode=143
Sep 17 19:19:39.463220 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.463146    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:39.463220 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.463206    2559 scope.go:117] "RemoveContainer" containerID="ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306"
Sep 17 19:19:39.466166 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:39.466139    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kjd2l\" is forbidden: User \"system:node:ip-10-0-143-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cs49d\": no relationship found between node 'ip-10-0-143-176.ec2.internal' and this object" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:39.471044 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.471021    2559 scope.go:117] "RemoveContainer" containerID="23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"
Sep 17 19:19:39.473237 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:39.473215    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kjd2l\" is forbidden: User \"system:node:ip-10-0-143-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cs49d\": no relationship found between node 'ip-10-0-143-176.ec2.internal' and this object" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" pod="openshift-must-gather-cs49d/must-gather-kjd2l"
Sep 17 19:19:39.482251 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.482230    2559 scope.go:117] "RemoveContainer" containerID="ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306"
Sep 17 19:19:39.482536 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:39.482514    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306\": container with ID starting with ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306 not found: ID does not exist" containerID="ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306"
Sep 17 19:19:39.482606 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.482547    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306"} err="failed to get container status \"ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306\": rpc error: code = NotFound desc = could not find container \"ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306\": container with ID starting with ec562570476c1891dc37ab9b5837e63fbeff218c82425dcffd055135fb01b306 not found: ID does not exist"
Sep 17 19:19:39.482606 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.482567    2559 scope.go:117] "RemoveContainer" containerID="23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"
Sep 17 19:19:39.483131 ip-10-0-143-176 kubenswrapper[2559]: E0917 19:19:39.483056    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039\": container with ID starting with 23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039 not found: ID does not exist" containerID="23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"
Sep 17 19:19:39.483131 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.483093    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039"} err="failed to get container status \"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039\": rpc error: code = NotFound desc = could not find container \"23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039\": container with ID starting with 23b129b1d67e84ce839a46965dd6cce013696e6d22ae5b35f58f932f5d83c039 not found: ID does not exist"
Sep 17 19:19:39.940139 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:39.940102    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" path="/var/lib/kubelet/pods/51cd86bc-bcbf-4ad8-9155-8a6249ded021/volumes"
Sep 17 19:19:40.775704 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:40.775671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:40.864554 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:40.864517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lmflg_34eb689d-dd14-41a6-a163-84c3650a3d1b/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:42.077469 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:42.077445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-n9xrd_d0ee7e92-446b-4fd2-817f-363399f71b8c/cluster-monitoring-operator/0.log"
Sep 17 19:19:42.394655 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:42.394566    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-656f8dbc68-lrlwz_a617db0e-a75a-475b-a60a-e9bee4e463e2/metrics-server/0.log"
Sep 17 19:19:42.737940 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:42.737852    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssxzd_abeac62d-87c5-4547-9768-ac33b6b70dad/node-exporter/0.log"
Sep 17 19:19:42.780144 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:42.780107    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssxzd_abeac62d-87c5-4547-9768-ac33b6b70dad/kube-rbac-proxy/0.log"
Sep 17 19:19:42.816940 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:42.816910    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssxzd_abeac62d-87c5-4547-9768-ac33b6b70dad/init-textfile/0.log"
Sep 17 19:19:43.135245 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.135123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/prometheus/0.log"
Sep 17 19:19:43.164947 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.164916    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/config-reloader/0.log"
Sep 17 19:19:43.221437 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.221405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/thanos-sidecar/0.log"
Sep 17 19:19:43.294589 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.294563    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/kube-rbac-proxy-web/0.log"
Sep 17 19:19:43.343987 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.343956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/kube-rbac-proxy/0.log"
Sep 17 19:19:43.381674 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.381646    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/kube-rbac-proxy-thanos/0.log"
Sep 17 19:19:43.425894 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:43.425864    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_34b7af9d-1e61-42c8-9b1c-6f5fe8b33b20/init-config-reloader/0.log"
Sep 17 19:19:44.778311 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778274    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"]
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778569    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="gather"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778579    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="gather"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778594    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778600    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778606    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778616    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kube-rbac-proxy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kube-rbac-proxy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="copy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778637    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="copy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778683    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cb3bb1-cb62-439a-98d2-10f2e02ec99a" containerName="switch-graph-29aad"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778690    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="copy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778696    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kube-rbac-proxy"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778701    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51cd86bc-bcbf-4ad8-9155-8a6249ded021" containerName="gather"
Sep 17 19:19:44.778721 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.778706    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ced5fb-aad4-4cb5-a6fa-d2c8800e62b5" containerName="kserve-container"
Sep 17 19:19:44.782733 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.782714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.785274 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.785254    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jrbn\"/\"kube-root-ca.crt\""
Sep 17 19:19:44.786124 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.786105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jrbn\"/\"openshift-service-ca.crt\""
Sep 17 19:19:44.786245 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.786108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2jrbn\"/\"default-dockercfg-4748h\""
Sep 17 19:19:44.792723 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.792695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"]
Sep 17 19:19:44.872705 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.872670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-sys\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.872906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.872715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-proc\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.872906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.872736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-podres\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.872906 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.872861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-lib-modules\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.873022 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.872907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dxk\" (UniqueName: \"kubernetes.io/projected/ca6f2611-981c-45ce-bfe4-84029c756c5c-kube-api-access-44dxk\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973480 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-lib-modules\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973682 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dxk\" (UniqueName: \"kubernetes.io/projected/ca6f2611-981c-45ce-bfe4-84029c756c5c-kube-api-access-44dxk\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973682 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-sys\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973682 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-proc\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973682 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-podres\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973682 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-lib-modules\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973929 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-sys\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973929 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-podres\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.973929 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.973700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca6f2611-981c-45ce-bfe4-84029c756c5c-proc\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:44.983744 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:44.983723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dxk\" (UniqueName: \"kubernetes.io/projected/ca6f2611-981c-45ce-bfe4-84029c756c5c-kube-api-access-44dxk\") pod \"perf-node-gather-daemonset-nf2rx\" (UID: \"ca6f2611-981c-45ce-bfe4-84029c756c5c\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:45.054778 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.054699    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-jx4ts_396a48fe-1cfd-478c-ae9a-3e2eb49d5017/networking-console-plugin/0.log"
Sep 17 19:19:45.092413 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.092377    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:45.363665 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.363582    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"]
Sep 17 19:19:45.367588 ip-10-0-143-176 kubenswrapper[2559]: W0917 19:19:45.367556    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podca6f2611_981c_45ce_bfe4_84029c756c5c.slice/crio-7b5326ecb099003334000e465cfafb84d66ab45153baac892f8fecfbff45593a WatchSource:0}: Error finding container 7b5326ecb099003334000e465cfafb84d66ab45153baac892f8fecfbff45593a: Status 404 returned error can't find the container with id 7b5326ecb099003334000e465cfafb84d66ab45153baac892f8fecfbff45593a
Sep 17 19:19:45.491799 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.491765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx" event={"ID":"ca6f2611-981c-45ce-bfe4-84029c756c5c","Type":"ContainerStarted","Data":"24ea1ee2a2d6fdf08baf9fc61872da9d22b8e35c9d17fe08155302062aa65256"}
Sep 17 19:19:45.491981 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.491823    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx" event={"ID":"ca6f2611-981c-45ce-bfe4-84029c756c5c","Type":"ContainerStarted","Data":"7b5326ecb099003334000e465cfafb84d66ab45153baac892f8fecfbff45593a"}
Sep 17 19:19:45.491981 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.491892    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:45.511697 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.511654    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx" podStartSLOduration=1.5116384360000001 podStartE2EDuration="1.511638436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:45.511045724 +0000 UTC m=+3028.061269610" watchObservedRunningTime="2026-09-17 19:19:45.511638436 +0000 UTC m=+3028.061862323"
Sep 17 19:19:45.546974 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.546950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/1.log"
Sep 17 19:19:45.555310 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:45.555283    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xcgc8_cc9bafcc-d25f-4e55-8f3d-5eb321bbde11/console-operator/2.log"
Sep 17 19:19:46.544782 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:46.544740    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-7qgvc_4fee1728-d0c7-452a-88da-1f97a178a7d1/volume-data-source-validator/0.log"
Sep 17 19:19:47.609645 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:47.609612    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hskxl_53bcfcd2-8b30-48bb-893f-74f7eefe20e8/dns/0.log"
Sep 17 19:19:47.647267 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:47.647237    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hskxl_53bcfcd2-8b30-48bb-893f-74f7eefe20e8/kube-rbac-proxy/0.log"
Sep 17 19:19:47.834757 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:47.834723    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lw8h_8fe2d461-8d80-4cb2-86fd-436d591bdafc/dns-node-resolver/0.log"
Sep 17 19:19:48.653692 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:48.653653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rl9dv_ca1cbb8d-74a1-4b94-82c2-90d6dbec2ed6/node-ca/0.log"
Sep 17 19:19:49.541420 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:49.541381    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d4456f768-xpfh4_3c6b2e61-d20e-4c70-98b4-03d8f5b84cd4/router/0.log"
Sep 17 19:19:49.992349 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:49.992270    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs4w8_75347877-8dc6-45b1-be39-82372a6459ec/serve-healthcheck-canary/0.log"
Sep 17 19:19:50.402609 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:50.402569    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gmcpd_caef9342-c962-42d4-b237-ed36f6d1a603/insights-operator/0.log"
Sep 17 19:19:50.403954 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:50.403928    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gmcpd_caef9342-c962-42d4-b237-ed36f6d1a603/insights-operator/1.log"
Sep 17 19:19:50.529364 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:50.529331    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwwfq_c5505393-5cb7-46ff-93ac-aa8400c4f92d/kube-rbac-proxy/0.log"
Sep 17 19:19:50.559761 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:50.559730    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwwfq_c5505393-5cb7-46ff-93ac-aa8400c4f92d/exporter/0.log"
Sep 17 19:19:50.589352 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:50.589310    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwwfq_c5505393-5cb7-46ff-93ac-aa8400c4f92d/extractor/0.log"
Sep 17 19:19:51.505514 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:51.505481    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-nf2rx"
Sep 17 19:19:53.082727 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:53.082698    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-kkjbb_75c467cf-84ff-485e-80f1-77d77d327d18/server/0.log"
Sep 17 19:19:59.455143 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:59.455106    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-gs7jx_c0d6e2ef-a0b6-4517-ae26-54d0803e3599/kube-storage-version-migrator-operator/1.log"
Sep 17 19:19:59.456816 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:19:59.456788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-gs7jx_c0d6e2ef-a0b6-4517-ae26-54d0803e3599/kube-storage-version-migrator-operator/0.log"
Sep 17 19:20:00.616193 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:00.616112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6mxw8_337523de-e49b-4985-9e89-8f641bdd018e/kube-multus/0.log"
Sep 17 19:20:00.900641 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:00.900554    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:20:00.934134 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:00.934106    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/egress-router-binary-copy/0.log"
Sep 17 19:20:00.962254 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:00.962224    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/cni-plugins/0.log"
Sep 17 19:20:00.989014 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:00.988990    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/bond-cni-plugin/0.log"
Sep 17 19:20:01.017697 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:01.017673    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/routeoverride-cni/0.log"
Sep 17 19:20:01.051542 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:01.051507    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/whereabouts-cni-bincopy/0.log"
Sep 17 19:20:01.078433 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:01.078408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fbrlc_cd2f3550-21bf-4f33-8a83-c2c84667900a/whereabouts-cni/0.log"
Sep 17 19:20:01.413636 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:01.413608    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mtf79_38b3f519-272f-4c6a-8cd1-f78ed4884548/network-metrics-daemon/0.log"
Sep 17 19:20:01.437816 ip-10-0-143-176 kubenswrapper[2559]: I0917 19:20:01.437790    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mtf79_38b3f519-272f-4c6a-8cd1-f78ed4884548/kube-rbac-proxy/0.log"