Jul 09 09:33:34.853497 ip-10-0-141-211 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.241717    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246967    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246976    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246981    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246984    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246987    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246989    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246992    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.298321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246995    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.246997    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247004    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247009    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247012    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247014    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247016    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247019    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247022    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247024    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247027    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247029    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247032    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247034    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247037    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247039    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247042    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247044    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247047    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247050    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.299827 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247052    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247055    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247058    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247061    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247063    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247066    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247068    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247071    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247074    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247076    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247079    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247081    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247083    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247086    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247088    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247091    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247094    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247098    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247101    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.300395 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247104    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247107    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247109    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247111    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247114    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247116    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247119    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247121    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247124    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247126    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247129    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247132    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247135    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247137    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247140    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247144    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247147    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247150    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247153    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.300925 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247157    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247160    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247162    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247165    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247168    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247170    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247173    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247175    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247194    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247197    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247201    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247203    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247206    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247210    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247213    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247215    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247218    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247220    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247223    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247225    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.301477 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247228    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247230    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247232    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247235    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247238    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247241    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247243    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247246    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247248    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247251    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247253    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247256    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247259    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.247262    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249225    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249237    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249241    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249244    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249247    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249250    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.457044 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249253    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.318907 ip-10-0-141-211 systemd[1]: Started Kubernetes Kubelet.
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249255    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249258    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249261    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249263    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249268    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249273    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249276    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249279    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249282    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249285    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249287    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249290    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249292    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249296    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249300    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249304    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249309    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249313    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249315    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.474176 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249318    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249325    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249328    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249331    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249333    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249336    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249340    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249342    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249345    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249348    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249351    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249353    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249356    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249358    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249361    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249363    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249367    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249369    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249372    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249375    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.475075 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249385    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249389    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249393    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249397    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249403    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249406    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249409    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249411    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249416    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249419    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249422    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249424    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249427    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249429    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249432    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249434    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249438    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249440    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249443    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249445    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.475950 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249448    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249450    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249454    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249457    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249459    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249462    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249465    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249469    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249473    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249477    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249481    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249485    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249487    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249490    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249492    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249496    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249499    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249503    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249506    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.476630 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249510    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249515    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249518    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249521    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249523    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249526    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249529    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249531    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249534    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249536    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249539    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249541    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249544    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249547    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249928    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249934    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249937    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249939    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249942    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.477749 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249945    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249949    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249954    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249958    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249962    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249966    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249970    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249974    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249977    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249980    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249983    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249987    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249990    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249992    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249997    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.249999    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250002    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250005    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250008    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.478428 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250010    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250017    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250019    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250022    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250024    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250027    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250032    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250036    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250040    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250044    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250048    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250051    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250053    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250056    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250058    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250061    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250065    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250068    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250070    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250073    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.479112 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250075    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250078    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250082    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250085    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250088    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250092    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250095    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250099    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250101    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250104    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250107    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250110    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250115    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250121    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250125    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250129    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250132    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250134    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250136    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.479746 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250139    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250141    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250144    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250146    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250149    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250151    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250153    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250156    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250158    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250161    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250163    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250166    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250168    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250172    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250174    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250190    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250194    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250198    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250202    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250207    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.480422 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250212    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250214    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250217    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250220    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250222    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250225    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250229    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250231    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250234    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250237    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250239    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250241    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250244    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250246    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250249    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250252    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250318    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250324    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250331    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250335    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250339    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250342    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 09:33:35.481036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250346    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250350    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250353    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250358    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250363    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250369    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250374    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250377    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250380    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250383    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250388    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250391    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250395    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250397    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250400    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250404    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250408    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250411    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250414    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250417    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250420    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250423    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250426    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250429    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 09:33:35.481834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250432    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250436    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250442    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250446    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250451    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250455    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250458    2576 flags.go:64] FLAG: --enable-server="true"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250461    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250466    2576 flags.go:64] FLAG: --event-burst="100"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250469    2576 flags.go:64] FLAG: --event-qps="50"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250471    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250474    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250477    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250481    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250484    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250486    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250489    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250492    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250496    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250500    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250503    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250505    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250508    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250511    2576 flags.go:64] FLAG: --feature-gates=""
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250517    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 09:33:35.482603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250522    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250526    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250531    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250536    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250539    2576 flags.go:64] FLAG: --help="false"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250542    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250545    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250548    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250551    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250554    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250558    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250560    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250563    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250566    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250569    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250571    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250574    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250577    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250580    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250582    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250585    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250588    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250591    2576 flags.go:64] FLAG: --lock-file=""
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250593    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 09:33:35.483448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250597    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250603    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250612    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250617    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250620    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250623    2576 flags.go:64] FLAG: --logging-format="text"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250626    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250630    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250633    2576 flags.go:64] FLAG: --manifest-url=""
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250636    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250640    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250643    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250647    2576 flags.go:64] FLAG: --max-pods="110"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250650    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250652    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250655    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250658    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250661    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250663    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250666    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250673    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250676    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250681    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 09:33:35.484123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250685    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250690    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250698    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250701    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250704    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250707    2576 flags.go:64] FLAG: --port="10250"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250710    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250712    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d10a14be7d19ba76"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250715    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250718    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250721    2576 flags.go:64] FLAG: --register-node="true"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250725    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250732    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250736    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250738    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250741    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250746    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250749    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250752    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250755    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250758    2576 flags.go:64] FLAG: --runonce="false"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250762    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250767    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250772    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250777    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 09:33:35.484858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250780    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250783    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250785    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250788    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250792    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250794    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250797    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250800    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250803    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250805    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250808    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250813    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250816    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250819    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250822    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250825    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250827    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250830    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250834    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250837    2576 flags.go:64] FLAG: --v="2"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250842    2576 flags.go:64] FLAG: --version="false"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250849    2576 flags.go:64] FLAG: --vmodule=""
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250855    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.250860    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250982    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.486107 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250987    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250990    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250993    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250996    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.250999    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251003    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251007    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251011    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251015    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251019    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251024    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251026    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251029    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251031    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251034    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251037    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251039    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251042    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251044    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.486894 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251047    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251050    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251052    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251056    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251060    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251063    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251067    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251070    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251073    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251076    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251079    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251084    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251089    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251093    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251098    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251102    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251105    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251107    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251110    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.650544 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251112    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251115    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251117    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251120    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251122    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251125    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251128    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251130    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251133    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251135    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251138    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251140    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251142    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251145    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251147    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251150    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251152    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251154    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251157    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.651678 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251161    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251164    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251167    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251171    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251175    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251196    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251201    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251203    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251206    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251209    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251211    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251215    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251218    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251221    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251223    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251226    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251229    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251231    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251234    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251237    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.652574 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251239    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251242    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251245    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251247    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251250    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251253    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251255    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251258    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251260    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251264    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251268    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251272    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251280    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251285    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251288    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251290    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251293    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251297    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251300    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251302    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.653272 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.251305    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.251310    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261000    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261016    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261028    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261036    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261089    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261094    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261097    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261100    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261103    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261106    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261108    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261111    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261114    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.653935 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261116    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261119    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261122    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261124    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261127    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261130    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261133    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261135    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261141    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261143    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261146    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261148    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261151    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261154    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261156    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261159    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261161    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261164    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261167    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261170    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.654586 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261173    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261176    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261195    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261199    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261202    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261207    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261210    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261213    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261216    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261219    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261222    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261225    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261228    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261231    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261234    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261236    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261239    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261241    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261244    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.655157 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261247    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261251    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261254    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261256    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261259    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261261    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261264    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261266    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261269    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261273    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261276    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261279    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261282    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261285    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261288    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261291    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261293    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261296    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261299    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.655753 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261301    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261304    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261307    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261310    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261312    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261314    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261317    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261319    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261322    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261324    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261327    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261330    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261332    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261335    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261339    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261341    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261343    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261347    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261350    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261353    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.656321 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261355    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261358    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261360    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261363    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261366    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261368    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261371    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261373    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261382    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261386    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261389    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261392    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261397    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261505    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261509    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:35.656972 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261513    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261515    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261518    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261521    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261523    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261526    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261529    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261531    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261534    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261536    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261539    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261543    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261546    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261548    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261551    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261555    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261557    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261560    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261562    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:35.657443 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261565    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261567    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261570    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261573    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261575    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261578    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261580    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261583    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261586    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261588    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261591    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261593    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261596    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261599    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261602    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261605    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261607    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261610    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261613    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261615    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:35.658018 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261618    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261620    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261622    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261626    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261632    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261635    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261639    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261641    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261646    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261649    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261652    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261655    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261658    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261661    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261663    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261666    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261668    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261671    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261673    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:35.658645 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261676    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261679    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261681    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261684    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261686    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261689    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261691    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261694    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261696    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261698    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261701    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261703    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261706    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261708    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261711    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261713    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261715    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261719    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261721    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261724    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:35.659235 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261727    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261731    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261734    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261736    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261739    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261742    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261746    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261749    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261752    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261754    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261757    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261760    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261763    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261766    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261769    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261771    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261774    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261777    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:35.659806 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:35.261779    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261784    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.261876    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.264128    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.265970    2576 server.go:1019] "Starting client certificate rotation"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.266071    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.266101    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.269535    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.270559    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.282252    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.284502    2576 log.go:25] "Validated CRI v1 image API"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.285257    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 09:33:35.660432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.289246    2576 fs.go:135] Filesystem UUIDs: map[47ff2bcf-4be7-4e37-ae08-4e765bdea9c8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b393863c-4462-467d-8c6e-f36885981da5:/dev/nvme0n1p4]
Jul 09 09:33:35.660805 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.289260    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.294104    2576 manager.go:216] Machine: {Timestamp:2026-07-09 09:33:35.292747406 +0000 UTC m=+0.328557102 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076722 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a3f9c6a52c1778c46bff5ec6654e8 SystemUUID:ec2a3f9c-6a52-c177-8c46-bff5ec6654e8 BootID:f9b50972-24d2-456e-9efe-29eb1460e766 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:ee:e5:72:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:ee:e5:72:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:a6:5e:54:b8:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.294207    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.294292    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.297194    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.297220    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.298026    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.298036    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 09:33:35.660918 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.298043    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.298798    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.300705    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.301091    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305144    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305276    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305301    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305310    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305333    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305345    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305356    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305675    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.305691    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.308052    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.310279    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.310306    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311721    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311734    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311750    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311756    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311766    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311775    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311786    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311791    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311797    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311802    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311818    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.311824    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.312623    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.312630    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.313778    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.313943    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.315490    2576 server.go:1268] "Started kubelet"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.315544    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.315581    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.315631    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.319172    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.319811    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.321214    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 09:33:35.663523 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.322088    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.328258    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-211.ec2.internal.18c0954e25f4361d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-211.ec2.internal,UID:ip-10-0-141-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-211.ec2.internal,},FirstTimestamp:2026-07-09 09:33:35.315465757 +0000 UTC m=+0.351275453,LastTimestamp:2026-07-09 09:33:35.315465757 +0000 UTC m=+0.351275453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-211.ec2.internal,}"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.329433    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.329443    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.329532    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330195    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330266    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330295    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330373    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330383    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330443    2576 factory.go:153] Registering CRI-O factory
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330455    2576 factory.go:223] Registration of the crio container factory successfully
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330500    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330507    2576 factory.go:55] Registering systemd factory
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330513    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330533    2576 factory.go:103] Registering Raw factory
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330545    2576 manager.go:1196] Started watching for new ooms in manager
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.330832    2576 manager.go:349] Starting recovery of all containers
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.333211    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.333326    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.335533    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xw7gw"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.340193    2576 manager.go:354] Recovery completed
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.344801    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.345659    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xw7gw"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.348569    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.348586    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.348596    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.349109    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.349116    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.349129    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.349725    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.350461    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.664860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.350494    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.350510    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.351062    2576 policy_none.go:49] "None policy: Start"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.351075    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.351085    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.353043    2576 policy_none.go:47] "Start"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.355441    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-211.ec2.internal.18c0954e27ed7ab8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-211.ec2.internal,UID:ip-10-0-141-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-211.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-211.ec2.internal,},FirstTimestamp:2026-07-09 09:33:35.348579 +0000 UTC m=+0.384388698,LastTimestamp:2026-07-09 09:33:35.348579 +0000 UTC m=+0.384388698,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-211.ec2.internal,}"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.398672    2576 manager.go:337] "Starting Device Plugin manager"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.398696    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.398705    2576 server.go:87] "Starting device plugin registration server"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.398917    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.399012    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.399108    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.399194    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.399203    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.399580    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.399627    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.499447    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.500301    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.500332    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.500345    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.500372    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.511156    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.511174    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-211.ec2.internal\": node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.533859    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.535127    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.535176    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.535200    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.535243    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.537479    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.543136    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.635476    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"]
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.635544    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.666088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.637727    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.637758    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.637771    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640011    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640174    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640831    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640857    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640874    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640893    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640911    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.640926    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.643171    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.643201    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.643239    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.644482    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.644507    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:35.667087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.644527    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.677975    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-211.ec2.internal\" not found" node="ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.694793    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-211.ec2.internal\" not found" node="ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.732212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.732253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.732275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/747907e14aa1fadee2058ca31d2de152-config\") pod \"kube-apiserver-proxy-ip-10-0-141-211.ec2.internal\" (UID: \"747907e14aa1fadee2058ca31d2de152\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.748817    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/747907e14aa1fadee2058ca31d2de152-config\") pod \"kube-apiserver-proxy-ip-10-0-141-211.ec2.internal\" (UID: \"747907e14aa1fadee2058ca31d2de152\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/747907e14aa1fadee2058ca31d2de152-config\") pod \"kube-apiserver-proxy-ip-10-0-141-211.ec2.internal\" (UID: \"747907e14aa1fadee2058ca31d2de152\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832941    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:35.840539 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.832997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f85054bc810612ea5f498fbc6f7b1c22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal\" (UID: \"f85054bc810612ea5f498fbc6f7b1c22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:36.009014 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.850225    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.009014 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:35.951254    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.009014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:35.987611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:36.009014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.005833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:36.186094 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:36.051861    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.186094 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:36.152299    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.214364    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:36.253090    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.265124    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.265301    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.265328    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.348916    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 09:28:35 +0000 UTC" deadline="2028-02-10 03:05:50.835619883 +0000 UTC"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.348945    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13937h32m14.486678694s"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:36.353204    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-211.ec2.internal\" not found"
Jul 09 09:33:36.386154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.377623    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:36.606654 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.431901    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:36.606654 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.444294    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 09:33:36.606654 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.445316    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal"
Jul 09 09:33:36.606654 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.454901    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 09:33:36.808664 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.785146    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:36.985352 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:36.822160    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747907e14aa1fadee2058ca31d2de152.slice/crio-83cf33ca28794dbf28b7bfacf98ec64252c600b7568a4054673a56bc4e536fe5 WatchSource:0}: Error finding container 83cf33ca28794dbf28b7bfacf98ec64252c600b7568a4054673a56bc4e536fe5: Status 404 returned error can't find the container with id 83cf33ca28794dbf28b7bfacf98ec64252c600b7568a4054673a56bc4e536fe5
Jul 09 09:33:36.985352 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:36.827135    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85054bc810612ea5f498fbc6f7b1c22.slice/crio-40692730e37849480c84a9e02d8362f528aa2b3c911a42913670db970ea29316 WatchSource:0}: Error finding container 40692730e37849480c84a9e02d8362f528aa2b3c911a42913670db970ea29316: Status 404 returned error can't find the container with id 40692730e37849480c84a9e02d8362f528aa2b3c911a42913670db970ea29316
Jul 09 09:33:36.985352 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:36.829280    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:33:37.333061 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.307975    2576 apiserver.go:52] "Watching apiserver"
Jul 09 09:33:37.333061 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.331043    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 09:33:37.345086 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.343669    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.363512    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.363976    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-29jzx","kube-system/konnectivity-agent-fsw2t","openshift-image-registry/node-ca-jcl8t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal","openshift-multus/multus-wghbx","openshift-network-diagnostics/network-check-target-nlkst","openshift-network-operator/iptables-alerter-s7xhc","kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd","openshift-cluster-node-tuning-operator/tuned-w5ckn","openshift-dns/node-resolver-xpdj2","openshift-multus/multus-additional-cni-plugins-b58d9","openshift-multus/network-metrics-daemon-ws58m","openshift-ovn-kubernetes/ovnkube-node-xdmzb"]
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.365180    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7f6zm"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.366815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.366889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.369079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.371342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.371996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.372059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sqpp8\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.372003    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.373610    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.373657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.374102    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.374170    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6ptw\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.374192    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.375394    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7f6zm"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.375850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.375914    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.376317    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.376437    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.376447    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.376513    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.376700    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4jjnp\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.378806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.381472    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w75b5\""
Jul 09 09:33:37.435717 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.381483    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.381574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.381774    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.381794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.383897    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.384244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.384357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.384659    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w5tb6\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.384836    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.386757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.387244    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.387297    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zb4dg\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.387301    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.388961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.389380    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.389989    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.390030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vvqv\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.391243    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.391441    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.391515    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xcrp4\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.391688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.391764    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.396282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399369    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399459    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399706    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399728    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399732    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsj7c\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399870    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.399917    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 09:33:37.439020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.431415    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-node-log\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-env-overrides\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-iptables-alerter-script\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.442985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-system-cni-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-multus\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-etc-kubernetes\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxjn\" (UniqueName: \"kubernetes.io/projected/8b33451b-4bff-453a-b2c4-0409582be7c7-kube-api-access-4bxjn\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-systemd-units\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-config\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.444812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-bin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-var-lib-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96crq\" (UniqueName: \"kubernetes.io/projected/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-kube-api-access-96crq\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-modprobe-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-sys\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-hostroot\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-os-release\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6hl\" (UniqueName: \"kubernetes.io/projected/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-kube-api-access-5s6hl\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-lib-modules\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-kubelet-config\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0df0c049-1ee6-423f-9523-2b81b60e9b71-konnectivity-ca\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-tuned\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.443547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-cnibin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.445949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-conf-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-kubelet\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-systemd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-netd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-dbus\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-host\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-k8s-cni-cncf-io\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76lt\" (UniqueName: \"kubernetes.io/projected/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-kube-api-access-s76lt\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0df0c049-1ee6-423f-9523-2b81b60e9b71-agent-certs\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-bin\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.449900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-device-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-daemon-config\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-host-slash\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-script-lib\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-registration-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-slash\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-socket-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-kubernetes\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-conf\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-tmp\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.444975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd672333-421c-432f-8bc5-972576520eca-tmp-dir\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.451026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-log-socket\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysconfig\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-os-release\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd672333-421c-432f-8bc5-972576520eca-hosts-file\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njddf\" (UniqueName: \"kubernetes.io/projected/53ce9f2d-7e9f-4933-aa70-236be2d8b270-kube-api-access-njddf\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-netns\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-var-lib-kubelet\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdjq\" (UniqueName: \"kubernetes.io/projected/83b54daa-e35f-4061-9647-4abe8e0a9591-kube-api-access-lrdjq\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-cni-binary-copy\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-netns\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.451954 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258r9\" (UniqueName: \"kubernetes.io/projected/fd672333-421c-432f-8bc5-972576520eca-kube-api-access-258r9\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-run\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-system-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-kubelet\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-ovn\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/719992de-6f8d-4661-bd86-c31a9f12e6b5-host\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/719992de-6f8d-4661-bd86-c31a9f12e6b5-kube-api-access-s7cll\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-sys-fs\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x598\" (UniqueName: \"kubernetes.io/projected/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kube-api-access-9x598\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-systemd\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-socket-dir-parent\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-etc-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/719992de-6f8d-4661-bd86-c31a9f12e6b5-serviceca\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.455219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-multus-certs\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.456362 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.445918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cnibin\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.649493 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-run\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.649699 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-system-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.649849 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-kubelet\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.649998 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-system-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.650157 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649767    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-run\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.650285 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-kubelet\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.650391 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.649982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-ovn\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.650518 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/719992de-6f8d-4661-bd86-c31a9f12e6b5-host\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.650591 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/719992de-6f8d-4661-bd86-c31a9f12e6b5-kube-api-access-s7cll\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.650591 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-sys-fs\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.650695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x598\" (UniqueName: \"kubernetes.io/projected/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kube-api-access-9x598\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.650695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/719992de-6f8d-4661-bd86-c31a9f12e6b5-host\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.650970 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-sys-fs\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.650970 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.650087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-ovn\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651115 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-systemd\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.651115 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-systemd\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.651232 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-socket-dir-parent\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651232 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-etc-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651354 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651295    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-etc-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651412 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-socket-dir-parent\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/719992de-6f8d-4661-bd86-c31a9f12e6b5-serviceca\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.651470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.651573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-multus-certs\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cnibin\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.651573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-node-log\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-env-overrides\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-iptables-alerter-script\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.651728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-system-cni-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-multus\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651867    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-multus\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-etc-kubernetes\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.651981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651931    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-node-log\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.651964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxjn\" (UniqueName: \"kubernetes.io/projected/8b33451b-4bff-453a-b2c4-0409582be7c7-kube-api-access-4bxjn\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-systemd-units\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652276    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/719992de-6f8d-4661-bd86-c31a9f12e6b5-serviceca\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cnibin\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-config\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.652402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652379    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 09:33:37.652672 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-system-cni-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.652672 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652524    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:37.652672 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.652663    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:37.652813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.652813 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.652769    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.152720103 +0000 UTC m=+3.188529806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:37.652931 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-etc-kubernetes\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.652931 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652914    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-bin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-var-lib-cni-bin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-systemd-units\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-iptables-alerter-script\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-multus-certs\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.652989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-var-lib-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653038 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96crq\" (UniqueName: \"kubernetes.io/projected/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-kube-api-access-96crq\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653049    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-var-lib-openvswitch\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-modprobe-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-sys\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-hostroot\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-sys\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-os-release\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6hl\" (UniqueName: \"kubernetes.io/projected/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-kube-api-access-5s6hl\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-lib-modules\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.653406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-kubelet-config\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0df0c049-1ee6-423f-9523-2b81b60e9b71-konnectivity-ca\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-tuned\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-cnibin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-conf-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-hostroot\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-kubelet\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-systemd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-modprobe-d\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-netd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.653736    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-os-release\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.653805    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.153786466 +0000 UTC m=+3.189596154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-cnibin\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.653883 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-conf-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-kubelet\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal" event={"ID":"747907e14aa1fadee2058ca31d2de152","Type":"ContainerStarted","Data":"83cf33ca28794dbf28b7bfacf98ec64252c600b7568a4054673a56bc4e536fe5"}
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-run-systemd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654395    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0df0c049-1ee6-423f-9523-2b81b60e9b71-konnectivity-ca\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-netd\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-kubelet-config\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.653737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-dbus\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.654579 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-host\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-k8s-cni-cncf-io\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-lib-modules\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76lt\" (UniqueName: \"kubernetes.io/projected/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-kube-api-access-s76lt\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0df0c049-1ee6-423f-9523-2b81b60e9b71-agent-certs\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-bin\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-device-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.654930 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.654887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.655815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-daemon-config\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.655815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-host-slash\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.655815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-script-lib\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-cni-bin\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-registration-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655888    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-host\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-slash\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-k8s-cni-cncf-io\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.655974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-socket-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.655995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-kubernetes\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-conf\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-socket-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-tmp\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd672333-421c-432f-8bc5-972576520eca-tmp-dir\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-log-socket\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.656307 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-daemon-config\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.656748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysconfig\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.656748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.656748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656497    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-os-release\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.656748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.656748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-device-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-multus-cni-dir\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysconfig\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656858    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-env-overrides\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-os-release\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-host-slash\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd672333-421c-432f-8bc5-972576520eca-hosts-file\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656932    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.656978 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.657392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.656986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njddf\" (UniqueName: \"kubernetes.io/projected/53ce9f2d-7e9f-4933-aa70-236be2d8b270-kube-api-access-njddf\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.657392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.657021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-netns\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.657392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.657054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-var-lib-kubelet\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.657392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.657112    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-var-lib-kubelet\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.657392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.657149    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd672333-421c-432f-8bc5-972576520eca-hosts-file\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.657728 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.657707    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.658167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658045    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-run-netns\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.658167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-script-lib\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.658167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdjq\" (UniqueName: \"kubernetes.io/projected/83b54daa-e35f-4061-9647-4abe8e0a9591-kube-api-access-lrdjq\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.658370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-cni-binary-copy\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.658370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-netns\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.658370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258r9\" (UniqueName: \"kubernetes.io/projected/fd672333-421c-432f-8bc5-972576520eca-kube-api-access-258r9\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.658518 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-host-slash\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.658518 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-registration-dir\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.658712 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.658671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovnkube-config\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.660136 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.660080    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd672333-421c-432f-8bc5-972576520eca-tmp-dir\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.660239 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.660091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/021994c4-fc8b-4b9c-b154-569016c95dfe-dbus\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:37.660541 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.660515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.661299 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.661277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-tmp\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.661494 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.661473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-kubernetes\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.661642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.661606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-tuned\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.661977 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.661955    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x598\" (UniqueName: \"kubernetes.io/projected/ea34cedf-0c18-4f6e-925c-62c4a9bf7756-kube-api-access-9x598\") pod \"aws-ebs-csi-driver-node-f8mdd\" (UID: \"ea34cedf-0c18-4f6e-925c-62c4a9bf7756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.662059 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.661979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.662120 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662107    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83b54daa-e35f-4061-9647-4abe8e0a9591-etc-sysctl-conf\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.662219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.662376 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-log-socket\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.662625 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.662606    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:37.662729 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.662718    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:37.662814 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.662805    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:37.663023 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b33451b-4bff-453a-b2c4-0409582be7c7-cni-binary-copy\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.663023 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b33451b-4bff-453a-b2c4-0409582be7c7-host-run-netns\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.663023 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.662752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/719992de-6f8d-4661-bd86-c31a9f12e6b5-kube-api-access-s7cll\") pod \"node-ca-jcl8t\" (UID: \"719992de-6f8d-4661-bd86-c31a9f12e6b5\") " pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.663301 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.663273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal" event={"ID":"f85054bc810612ea5f498fbc6f7b1c22","Type":"ContainerStarted","Data":"40692730e37849480c84a9e02d8362f528aa2b3c911a42913670db970ea29316"}
Jul 09 09:33:37.663373 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:37.663288    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.163265814 +0000 UTC m=+3.199075512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:37.663831 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.663807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.667741 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.667681    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6hl\" (UniqueName: \"kubernetes.io/projected/471543cd-d0be-4b9a-8c44-b9d21c91a3a5-kube-api-access-5s6hl\") pod \"multus-additional-cni-plugins-b58d9\" (UID: \"471543cd-d0be-4b9a-8c44-b9d21c91a3a5\") " pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.668776 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.668731    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdjq\" (UniqueName: \"kubernetes.io/projected/83b54daa-e35f-4061-9647-4abe8e0a9591-kube-api-access-lrdjq\") pod \"tuned-w5ckn\" (UID: \"83b54daa-e35f-4061-9647-4abe8e0a9591\") " pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.669955 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.669903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258r9\" (UniqueName: \"kubernetes.io/projected/fd672333-421c-432f-8bc5-972576520eca-kube-api-access-258r9\") pod \"node-resolver-xpdj2\" (UID: \"fd672333-421c-432f-8bc5-972576520eca\") " pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.670446 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.670423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96crq\" (UniqueName: \"kubernetes.io/projected/46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a-kube-api-access-96crq\") pod \"ovnkube-node-xdmzb\" (UID: \"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.670554 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.670530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0df0c049-1ee6-423f-9523-2b81b60e9b71-agent-certs\") pod \"konnectivity-agent-fsw2t\" (UID: \"0df0c049-1ee6-423f-9523-2b81b60e9b71\") " pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.671087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.671062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxjn\" (UniqueName: \"kubernetes.io/projected/8b33451b-4bff-453a-b2c4-0409582be7c7-kube-api-access-4bxjn\") pod \"multus-wghbx\" (UID: \"8b33451b-4bff-453a-b2c4-0409582be7c7\") " pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.673624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.673597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njddf\" (UniqueName: \"kubernetes.io/projected/53ce9f2d-7e9f-4933-aa70-236be2d8b270-kube-api-access-njddf\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:37.674789 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.674746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76lt\" (UniqueName: \"kubernetes.io/projected/53793e29-e8b5-4732-acf1-9bc9f65b7ec0-kube-api-access-s76lt\") pod \"iptables-alerter-s7xhc\" (UID: \"53793e29-e8b5-4732-acf1-9bc9f65b7ec0\") " pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.681866 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.681847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:33:37.694209 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.694170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jcl8t"
Jul 09 09:33:37.697111 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.697085    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df0c049_1ee6_423f_9523_2b81b60e9b71.slice/crio-3fb62bf2256ab4bbc349b8d46274e3502ab9885416371d23f07f2becac57e64e WatchSource:0}: Error finding container 3fb62bf2256ab4bbc349b8d46274e3502ab9885416371d23f07f2becac57e64e: Status 404 returned error can't find the container with id 3fb62bf2256ab4bbc349b8d46274e3502ab9885416371d23f07f2becac57e64e
Jul 09 09:33:37.705591 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.705570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wghbx"
Jul 09 09:33:37.705798 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.705778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719992de_6f8d_4661_bd86_c31a9f12e6b5.slice/crio-40e5057a56e20720948121cf4410b7589dac8a427908d443e9d1ebc9a2b559fe WatchSource:0}: Error finding container 40e5057a56e20720948121cf4410b7589dac8a427908d443e9d1ebc9a2b559fe: Status 404 returned error can't find the container with id 40e5057a56e20720948121cf4410b7589dac8a427908d443e9d1ebc9a2b559fe
Jul 09 09:33:37.713584 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.713557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s7xhc"
Jul 09 09:33:37.718027 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.718003    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33451b_4bff_453a_b2c4_0409582be7c7.slice/crio-13ac4641bc4578fff4f5cb5ab5340f41f02129988237b8fbbe33efb80e3f0e2c WatchSource:0}: Error finding container 13ac4641bc4578fff4f5cb5ab5340f41f02129988237b8fbbe33efb80e3f0e2c: Status 404 returned error can't find the container with id 13ac4641bc4578fff4f5cb5ab5340f41f02129988237b8fbbe33efb80e3f0e2c
Jul 09 09:33:37.723743 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.723645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd"
Jul 09 09:33:37.728100 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.728075    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53793e29_e8b5_4732_acf1_9bc9f65b7ec0.slice/crio-ed3f4129edd2842c548706b5909c5423aa53892626091bd167442589a94aa5e1 WatchSource:0}: Error finding container ed3f4129edd2842c548706b5909c5423aa53892626091bd167442589a94aa5e1: Status 404 returned error can't find the container with id ed3f4129edd2842c548706b5909c5423aa53892626091bd167442589a94aa5e1
Jul 09 09:33:37.731065 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.730909    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5ckn"
Jul 09 09:33:37.734542 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.734514    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea34cedf_0c18_4f6e_925c_62c4a9bf7756.slice/crio-d7d2a8bc7d734264462a8a21892239abf5e2d4e9334739880c57ca3d3c72c628 WatchSource:0}: Error finding container d7d2a8bc7d734264462a8a21892239abf5e2d4e9334739880c57ca3d3c72c628: Status 404 returned error can't find the container with id d7d2a8bc7d734264462a8a21892239abf5e2d4e9334739880c57ca3d3c72c628
Jul 09 09:33:37.741955 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.741933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpdj2"
Jul 09 09:33:37.745308 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.745283    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b54daa_e35f_4061_9647_4abe8e0a9591.slice/crio-366963505a787391e682d64207de0c0bf2d74540ac2236091dcfa5d23c590704 WatchSource:0}: Error finding container 366963505a787391e682d64207de0c0bf2d74540ac2236091dcfa5d23c590704: Status 404 returned error can't find the container with id 366963505a787391e682d64207de0c0bf2d74540ac2236091dcfa5d23c590704
Jul 09 09:33:37.748149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.747981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b58d9"
Jul 09 09:33:37.755008 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:37.754874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:33:37.755181 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.755155    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd672333_421c_432f_8bc5_972576520eca.slice/crio-df51cd680d2eed54eefd643d528eb946eb1427471e053fcc9b0e055a62692201 WatchSource:0}: Error finding container df51cd680d2eed54eefd643d528eb946eb1427471e053fcc9b0e055a62692201: Status 404 returned error can't find the container with id df51cd680d2eed54eefd643d528eb946eb1427471e053fcc9b0e055a62692201
Jul 09 09:33:37.761747 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.761722    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471543cd_d0be_4b9a_8c44_b9d21c91a3a5.slice/crio-e020ddb6dd099aad3dd6522e10d75ce289690ada6820d088d2f6d54f99961001 WatchSource:0}: Error finding container e020ddb6dd099aad3dd6522e10d75ce289690ada6820d088d2f6d54f99961001: Status 404 returned error can't find the container with id e020ddb6dd099aad3dd6522e10d75ce289690ada6820d088d2f6d54f99961001
Jul 09 09:33:37.767227 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:33:37.767202    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bd2fa2_68cf_428d_9ef7_7e0ed0d77e5a.slice/crio-1c7e8ff92b92c152e9eebf2b141f9ad8950d1cc8c6fbbef96aab5f11b04644b2 WatchSource:0}: Error finding container 1c7e8ff92b92c152e9eebf2b141f9ad8950d1cc8c6fbbef96aab5f11b04644b2: Status 404 returned error can't find the container with id 1c7e8ff92b92c152e9eebf2b141f9ad8950d1cc8c6fbbef96aab5f11b04644b2
Jul 09 09:33:38.162754 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.162721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:38.162931 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.162770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:38.162931 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.162886    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:38.162931 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.162889    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:38.163064 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.162950    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:39.162931467 +0000 UTC m=+4.198741157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:38.163064 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.162967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:33:39.162959137 +0000 UTC m=+4.198768822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:38.263368 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.263333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:38.263539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.263467    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:38.263539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.263492    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:38.263539 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.263504    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:38.263701 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:38.263584    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:39.263549676 +0000 UTC m=+4.299359376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:38.378511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.378394    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:37 +0000 UTC" deadline="2028-01-16 12:54:44.80781296 +0000 UTC"
Jul 09 09:33:38.378511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.378456    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13347h21m6.429362148s"
Jul 09 09:33:38.667624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.667540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5ckn" event={"ID":"83b54daa-e35f-4061-9647-4abe8e0a9591","Type":"ContainerStarted","Data":"366963505a787391e682d64207de0c0bf2d74540ac2236091dcfa5d23c590704"}
Jul 09 09:33:38.669693 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.669662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s7xhc" event={"ID":"53793e29-e8b5-4732-acf1-9bc9f65b7ec0","Type":"ContainerStarted","Data":"ed3f4129edd2842c548706b5909c5423aa53892626091bd167442589a94aa5e1"}
Jul 09 09:33:38.671427 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.671352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wghbx" event={"ID":"8b33451b-4bff-453a-b2c4-0409582be7c7","Type":"ContainerStarted","Data":"13ac4641bc4578fff4f5cb5ab5340f41f02129988237b8fbbe33efb80e3f0e2c"}
Jul 09 09:33:38.673687 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.673598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd" event={"ID":"ea34cedf-0c18-4f6e-925c-62c4a9bf7756","Type":"ContainerStarted","Data":"d7d2a8bc7d734264462a8a21892239abf5e2d4e9334739880c57ca3d3c72c628"}
Jul 09 09:33:38.676361 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.676333    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jcl8t" event={"ID":"719992de-6f8d-4661-bd86-c31a9f12e6b5","Type":"ContainerStarted","Data":"40e5057a56e20720948121cf4410b7589dac8a427908d443e9d1ebc9a2b559fe"}
Jul 09 09:33:38.679358 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.679327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fsw2t" event={"ID":"0df0c049-1ee6-423f-9523-2b81b60e9b71","Type":"ContainerStarted","Data":"3fb62bf2256ab4bbc349b8d46274e3502ab9885416371d23f07f2becac57e64e"}
Jul 09 09:33:38.680816 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.680754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"1c7e8ff92b92c152e9eebf2b141f9ad8950d1cc8c6fbbef96aab5f11b04644b2"}
Jul 09 09:33:38.682518 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.682454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerStarted","Data":"e020ddb6dd099aad3dd6522e10d75ce289690ada6820d088d2f6d54f99961001"}
Jul 09 09:33:38.684262 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.684231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpdj2" event={"ID":"fd672333-421c-432f-8bc5-972576520eca","Type":"ContainerStarted","Data":"df51cd680d2eed54eefd643d528eb946eb1427471e053fcc9b0e055a62692201"}
Jul 09 09:33:38.800890 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:38.800555    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:39.171364 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.171327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:39.171598 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.171390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:39.171598 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.171516    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:39.171776 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.171607    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:41.171587713 +0000 UTC m=+6.207397399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:39.172273 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.172155    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:39.172273 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.172223    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:33:41.172204359 +0000 UTC m=+6.208014046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:39.272793 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.272701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:39.273099 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.272891    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:39.273099 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.272914    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:39.273099 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.272928    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:39.273099 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.272991    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:41.272971144 +0000 UTC m=+6.308780847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:39.378906 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.378827    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:37 +0000 UTC" deadline="2028-01-06 04:31:06.135594767 +0000 UTC"
Jul 09 09:33:39.378906 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.378864    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13098h57m26.756734853s"
Jul 09 09:33:39.536526 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.536390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:39.536683 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.536584    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:39.537012 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.536842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:39.537012 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.536961    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:39.537012 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:39.536997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:39.537252 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:39.537082    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:41.190334 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.190287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:41.190832 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.190399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:41.190832 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.190523    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:41.190832 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.190583    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.190564863 +0000 UTC m=+10.226374551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:41.191008 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.190995    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:41.191061 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.191045    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.191029416 +0000 UTC m=+10.226839106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:41.291657 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.291619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:41.291827 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.291802    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:41.291827 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.291821    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:41.291971 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.291834    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:41.291971 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.291883    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.291868146 +0000 UTC m=+10.327677847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:41.537169 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.536348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:41.537169 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.536495    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:41.537169 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.536941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:41.537169 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.537047    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:41.537797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:41.537628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:41.537797 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:41.537752    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:43.536302 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:43.536230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:43.536720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:43.536244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:43.536720 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:43.536364    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:43.536720 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:43.536459    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:43.536720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:43.536245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:43.536720 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:43.536570    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:45.226332 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.226295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:45.226772 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.226354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:45.226772 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.226505    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:45.226772 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.226563    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:53.226545092 +0000 UTC m=+18.262354778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:45.226772 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.226505    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:45.226772 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.226634    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:33:53.226620113 +0000 UTC m=+18.262429810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:45.327583 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.327544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:45.327759 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.327731    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:45.327759 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.327758    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:45.327870 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.327771    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:45.327870 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.327829    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:53.327809761 +0000 UTC m=+18.363619459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:45.536898 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.536785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:45.537043 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.536914    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:45.537043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.537033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:45.537147 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:45.537088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:45.537227 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.537191    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:45.537320 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:45.537297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:47.536626 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:47.536494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:47.536626 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:47.536615    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:47.537219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:47.536640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:47.537219 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:47.536765    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:47.537219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:47.536814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:47.537219 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:47.536893    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:49.535545 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:49.535475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:49.535545 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:49.535512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:49.536050 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:49.535612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:49.536050 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:49.535617    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:49.536050 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:49.535732    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:49.536050 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:49.535808    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:51.536327 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:51.536191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:51.536327 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:51.536190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:51.536327 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:51.536325    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:51.536879 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:51.536433    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:51.536879 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:51.536481    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:51.536879 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:51.536537    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:53.283765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.283562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:53.284241 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.283781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:53.284241 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.283713    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:53.284241 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.283873    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:53.284241 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.283907    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:34:09.283885101 +0000 UTC m=+34.319694790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:53.284241 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.283923    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:09.283916384 +0000 UTC m=+34.319726070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:53.384773 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.384746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:53.384942 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.384887    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:53.384942 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.384910    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:53.384942 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.384922    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:53.385099 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.384982    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:09.384964781 +0000 UTC m=+34.420774483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:53.535477 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.535382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:53.535477 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.535419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:53.535677 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.535511    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:53.535815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:53.535770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:53.535884 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.535861    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:53.535966 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:53.535941    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:55.537310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:55.537272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:55.537310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:55.537306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:55.537828 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:55.537272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:55.537828 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:55.537380    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:55.537828 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:55.537477    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:55.537828 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:55.537558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:57.535902 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.535874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:57.536401 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.535906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:57.536401 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.535884    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:57.536401 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:57.535993    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:57.536401 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:57.536093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:57.536401 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:57.536174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:57.740631 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.740409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5ckn" event={"ID":"83b54daa-e35f-4061-9647-4abe8e0a9591","Type":"ContainerStarted","Data":"e642dbce829568693349bf14e8726d9d1b8ca8a08e96d643bc820b311c7b63e8"}
Jul 09 09:33:57.744027 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.743948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wghbx" event={"ID":"8b33451b-4bff-453a-b2c4-0409582be7c7","Type":"ContainerStarted","Data":"fe1e739447daa0aaa4b3ec0467fd8b88b1ed381c76d57032925f063d3f98ff46"}
Jul 09 09:33:57.746399 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.746285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal" event={"ID":"747907e14aa1fadee2058ca31d2de152","Type":"ContainerStarted","Data":"ef627830d997d931cf07c2e631847b6858f0b065731e0c2a26d64786b3a38a37"}
Jul 09 09:33:57.748646 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.748422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"c2f5a3c9b621d96bb280343e95238841c20e6be4a5f4fd72597b1db74785d8f8"}
Jul 09 09:33:57.748646 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.748502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"77e9a2599dafda5a06a387a95ea1c0befb50195d70f8cecd36bc6edd62e7a0e6"}
Jul 09 09:33:57.785088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.785004    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-211.ec2.internal" podStartSLOduration=21.784983945 podStartE2EDuration="21.784983945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.784759357 +0000 UTC m=+22.820569061" watchObservedRunningTime="2026-07-09 09:33:57.784983945 +0000 UTC m=+22.820793650"
Jul 09 09:33:57.785287 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.785243    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w5ckn" podStartSLOduration=2.242434343 podStartE2EDuration="21.785231365s" totalImagesPullingTime="19.542797022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.765022383 +0000 UTC m=+22.800832089" watchObservedRunningTime="2026-07-09 09:33:57.785231365 +0000 UTC m=+22.821041071"
Jul 09 09:33:57.810024 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:57.809984    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wghbx" podStartSLOduration=2.229144854 podStartE2EDuration="21.809969676s" totalImagesPullingTime="19.580824822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.809511087 +0000 UTC m=+22.845320793" watchObservedRunningTime="2026-07-09 09:33:57.809969676 +0000 UTC m=+22.845779381"
Jul 09 09:33:58.752592 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.752385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jcl8t" event={"ID":"719992de-6f8d-4661-bd86-c31a9f12e6b5","Type":"ContainerStarted","Data":"d9f805c27412694b79ff6a372dda0549cef4350d9085f56519f90f0887e0f976"}
Jul 09 09:33:58.754201 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.754161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fsw2t" event={"ID":"0df0c049-1ee6-423f-9523-2b81b60e9b71","Type":"ContainerStarted","Data":"4264968314fbf588a638a987897c1da4bfb74669838a05cbcadc1f527bd4b4eb"}
Jul 09 09:33:58.757679 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.757656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"03256fc3230293ca06cb95ce24413cebf896fad31ef887faf0b389c23cbb9fe4"}
Jul 09 09:33:58.757777 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.757720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"063d58770a526b925273abebce9b31fc504ebbe134fc299b948fa3bc03ac31d1"}
Jul 09 09:33:58.757777 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.757751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"1d74534b0fe844c887f587f92992050b174f13933d4d8b04b5225df53fb95a1e"}
Jul 09 09:33:58.757889 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.757781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"f1f46a1f0870f06a90cf6970e70e4afed249b345a3a24be6ab0991b58342f72c"}
Jul 09 09:33:58.759376 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.759351    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="20973bc75284f2bc8484f92609ce4b61685bf207418923af660ef2b19fdcbed2" exitCode=0
Jul 09 09:33:58.759480 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.759383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"20973bc75284f2bc8484f92609ce4b61685bf207418923af660ef2b19fdcbed2"}
Jul 09 09:33:58.761174 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.761154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpdj2" event={"ID":"fd672333-421c-432f-8bc5-972576520eca","Type":"ContainerStarted","Data":"7b3e5e6343066042f8c87a4a453bba568fa7b7810ccb8c118e1b10314f6eb25d"}
Jul 09 09:33:58.762901 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.762879    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s7xhc" event={"ID":"53793e29-e8b5-4732-acf1-9bc9f65b7ec0","Type":"ContainerStarted","Data":"21efe512fbddb8ef2af91b214c96182b3b57e8d1f6ecab80e2fa15fb4066e6c7"}
Jul 09 09:33:58.764508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.764484    2576 generic.go:356] "Generic (PLEG): container finished" podID="f85054bc810612ea5f498fbc6f7b1c22" containerID="f24f2560e3527408d9f2c5ddad1e275aa0ab6162b81b8fd9036b44e4b09a8857" exitCode=0
Jul 09 09:33:58.764601 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.764571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal" event={"ID":"f85054bc810612ea5f498fbc6f7b1c22","Type":"ContainerDied","Data":"f24f2560e3527408d9f2c5ddad1e275aa0ab6162b81b8fd9036b44e4b09a8857"}
Jul 09 09:33:58.766420 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.766392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd" event={"ID":"ea34cedf-0c18-4f6e-925c-62c4a9bf7756","Type":"ContainerStarted","Data":"83a6d5601aaf98edeecc2f6ef574bf0720a148fe89ddff992484ce81eecdfdef"}
Jul 09 09:33:58.776871 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.776826    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jcl8t" podStartSLOduration=3.21566751 podStartE2EDuration="22.776810142s" totalImagesPullingTime="19.561142632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.776555426 +0000 UTC m=+23.812365133" watchObservedRunningTime="2026-07-09 09:33:58.776810142 +0000 UTC m=+23.812619847"
Jul 09 09:33:58.796594 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.796551    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s7xhc" podStartSLOduration=11.569670662 podStartE2EDuration="22.796538016s" totalImagesPullingTime="11.226867354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.79576181 +0000 UTC m=+23.831571516" watchObservedRunningTime="2026-07-09 09:33:58.796538016 +0000 UTC m=+23.832347722"
Jul 09 09:33:58.818441 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.818341    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xpdj2" podStartSLOduration=3.668626804 podStartE2EDuration="22.818323632s" totalImagesPullingTime="19.149696828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.81787332 +0000 UTC m=+23.853683026" watchObservedRunningTime="2026-07-09 09:33:58.818323632 +0000 UTC m=+23.854133337"
Jul 09 09:33:58.895422 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:58.895382    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fsw2t" podStartSLOduration=13.33042136 podStartE2EDuration="22.895365482s" totalImagesPullingTime="9.564944122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.895071078 +0000 UTC m=+23.930880783" watchObservedRunningTime="2026-07-09 09:33:58.895365482 +0000 UTC m=+23.931175187"
Jul 09 09:33:59.287361 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.287329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:33:59.511034 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.511011    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 09:33:59.535939 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.535916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:33:59.535939 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.535932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:33:59.536155 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.535937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:33:59.536155 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:59.536033    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:33:59.536155 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:59.536124    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:33:59.536333 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:33:59.536233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:33:59.770942 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.770915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal" event={"ID":"f85054bc810612ea5f498fbc6f7b1c22","Type":"ContainerStarted","Data":"137df7e8614d62d052d768851e52ff09e63b1535a06f0b40fbc0386757f95f90"}
Jul 09 09:33:59.773054 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.773030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd" event={"ID":"ea34cedf-0c18-4f6e-925c-62c4a9bf7756","Type":"ContainerStarted","Data":"f98b114f25190080480a6f38166ea3fd873cc608c7a000dd373442062b75b384"}
Jul 09 09:33:59.788730 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.788681    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-211.ec2.internal" podStartSLOduration=23.788662392 podStartE2EDuration="23.788662392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:59.787331452 +0000 UTC m=+24.823141148" watchObservedRunningTime="2026-07-09 09:33:59.788662392 +0000 UTC m=+24.824472098"
Jul 09 09:33:59.881158 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:33:59.881128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:00.284923 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.284894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:00.413492 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.413395    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T09:33:59.51103121Z","UUID":"8ecf5d79-11ff-42f2-b8de-7fff43c4215c","Handler":null,"Name":"","Endpoint":""}
Jul 09 09:34:00.415031 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.415007    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 09:34:00.415195 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.415048    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 09:34:00.485571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.485500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:00.683078 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.683042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:00.780166 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:00.780138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"706a2abbac6c4914f881e27c8bb735bd088f99568cf075c424f873a9439b82c3"}
Jul 09 09:34:01.088096 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.088015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:01.513879 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.513841    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:01.535909 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.535875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:01.535909 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.535893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:01.536164 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.535875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:01.536164 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:01.535999    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:01.536164 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:01.536098    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:01.536333 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:01.536224    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:01.555261 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.555233    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:34:01.555901 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.555877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:34:01.606606 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.606575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:01.665668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.665639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:01.767123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.767050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:01.785515 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.785479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd" event={"ID":"ea34cedf-0c18-4f6e-925c-62c4a9bf7756","Type":"ContainerStarted","Data":"6d456adbce5ba31d2b06f16ce1df5cec6752d52a54ef437779f1e709a4509ec2"}
Jul 09 09:34:01.786149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.785748    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:34:01.786316 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.786294    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fsw2t"
Jul 09 09:34:01.805279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:01.805230    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8mdd" podStartSLOduration=4.026181332 podStartE2EDuration="25.805215661s" totalImagesPullingTime="21.779034329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:01.804907928 +0000 UTC m=+26.840717634" watchObservedRunningTime="2026-07-09 09:34:01.805215661 +0000 UTC m=+26.841025366"
Jul 09 09:34:02.282158 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:02.282104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:02.478735 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:02.478701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:02.903439 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:02.903411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:03.108340 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.108287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:03.536380 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.536352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:03.536486 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.536352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:03.536537 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:03.536487    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:03.536537 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.536358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:03.536537 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:03.536518    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:03.536670 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:03.536602    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:03.757543 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.757521    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:03.793204 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.793167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" event={"ID":"46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a","Type":"ContainerStarted","Data":"85c105a8ac7cfa156ac2aa78856a1e2ec6262b728462e2f84cb942ff178d382d"}
Jul 09 09:34:03.793426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.793405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:03.795044 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.795024    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="10ebc37215ca61b21fc3313e76614d067ddc345a7525812f60e229f13bed4149" exitCode=0
Jul 09 09:34:03.795137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.795096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"10ebc37215ca61b21fc3313e76614d067ddc345a7525812f60e229f13bed4149"}
Jul 09 09:34:03.808273 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.808255    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:03.821332 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:03.821297    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb" podStartSLOduration=8.054656239 podStartE2EDuration="27.821285992s" totalImagesPullingTime="19.766629753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:03.819606961 +0000 UTC m=+28.855416677" watchObservedRunningTime="2026-07-09 09:34:03.821285992 +0000 UTC m=+28.857095696"
Jul 09 09:34:04.797987 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:04.797959    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:04.798743 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:04.798047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:04.812572 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:04.812546    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:05.536751 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:05.536729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:05.536874 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:05.536799    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:05.536921 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:05.536900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:05.537015 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:05.536996    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:05.537056 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:05.537045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:05.537108 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:05.537096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:05.801872 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:05.801813    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="872243a2f2916bee10fc12dacde44c1693f45099d2010419dbdd8df6324bd0f8" exitCode=0
Jul 09 09:34:05.802232 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:05.801882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"872243a2f2916bee10fc12dacde44c1693f45099d2010419dbdd8df6324bd0f8"}
Jul 09 09:34:07.535862 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:07.535700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:07.536287 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:07.535706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:07.536287 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:07.535969    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:07.536287 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:07.535708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:07.536287 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:07.536001    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:07.536287 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:07.536067    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:07.807856 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:07.807781    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="660da885df66510db562176447c15b48eaf3cb54bd93cb17b8a8ae8c642ab420" exitCode=0
Jul 09 09:34:07.807856 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:07.807820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"660da885df66510db562176447c15b48eaf3cb54bd93cb17b8a8ae8c642ab420"}
Jul 09 09:34:09.314967 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.314929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:09.315470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.315027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:09.315470 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.315098    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:34:09.315470 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.315149    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:34:09.315470 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.315197    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs podName:53ce9f2d-7e9f-4933-aa70-236be2d8b270 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:41.315163031 +0000 UTC m=+66.350972727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs") pod "network-metrics-daemon-ws58m" (UID: "53ce9f2d-7e9f-4933-aa70-236be2d8b270") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:34:09.315470 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.315218    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret podName:021994c4-fc8b-4b9c-b154-569016c95dfe nodeName:}" failed. No retries permitted until 2026-07-09 09:34:41.315207697 +0000 UTC m=+66.351017387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret") pod "global-pull-secret-syncer-29jzx" (UID: "021994c4-fc8b-4b9c-b154-569016c95dfe") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:34:09.415943 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.415911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:09.416116 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.416097    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:34:09.416162 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.416129    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:34:09.416162 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.416144    2576 projected.go:196] Error preparing data for projected volume kube-api-access-jgf4t for pod openshift-network-diagnostics/network-check-target-nlkst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:34:09.416285 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.416271    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t podName:fb2bea31-b084-4006-9ea1-02113a6eeb22 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:41.416250573 +0000 UTC m=+66.452060259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgf4t" (UniqueName: "kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t") pod "network-check-target-nlkst" (UID: "fb2bea31-b084-4006-9ea1-02113a6eeb22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:34:09.536284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.536250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:09.536284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.536285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:09.536511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:09.536250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:09.536511 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.536372    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:09.536511 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.536455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:09.536657 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:09.536533    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:11.535508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:11.535467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:11.535508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:11.535486    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:11.536093 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:11.535660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:11.536093 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:11.535657    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:11.536093 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:11.535767    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:11.536093 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:11.535846    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:11.819949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:11.819883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerStarted","Data":"8a7d167be7d15310b5af9441c0a09011d6c6eb5fed2e20fe84e140842dc3743b"}
Jul 09 09:34:12.774587 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:12.774551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:12.824908 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:12.824880    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="8a7d167be7d15310b5af9441c0a09011d6c6eb5fed2e20fe84e140842dc3743b" exitCode=0
Jul 09 09:34:12.825057 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:12.824925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"8a7d167be7d15310b5af9441c0a09011d6c6eb5fed2e20fe84e140842dc3743b"}
Jul 09 09:34:12.892404 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:12.892379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:12.996232 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:12.996204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:13.100298 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.100246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:13.190121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.190103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:13.535614 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.535588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:13.535614 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.535604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:13.535846 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.535604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:13.535846 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:13.535702    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:13.535846 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:13.535776    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:13.535982 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:13.535844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:13.830265 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.830196    2576 generic.go:356] "Generic (PLEG): container finished" podID="471543cd-d0be-4b9a-8c44-b9d21c91a3a5" containerID="05fa958d202499fbe31bd6f16b39d3cdda2a4593bcec035a5ea4dadc339e9ff7" exitCode=0
Jul 09 09:34:13.830265 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.830252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerDied","Data":"05fa958d202499fbe31bd6f16b39d3cdda2a4593bcec035a5ea4dadc339e9ff7"}
Jul 09 09:34:13.857968 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.857923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:13.958322 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:13.958298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:14.130919 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.130858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:14.163605 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.163573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlkst"]
Jul 09 09:34:14.163729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.163674    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:14.163786 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:14.163759    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:14.165695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.165676    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ws58m"]
Jul 09 09:34:14.165803 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.165771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:14.165900 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:14.165882    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:14.166429 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.166405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29jzx"]
Jul 09 09:34:14.166529 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.166492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:14.166613 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:14.166582    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:14.216143 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.216111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:14.836043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.835821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b58d9" event={"ID":"471543cd-d0be-4b9a-8c44-b9d21c91a3a5","Type":"ContainerStarted","Data":"fe3a0d63da8b23f658bcf7c662494bd6a901485594e3fe2d6a384f09e672fcc5"}
Jul 09 09:34:14.868284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.868241    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b58d9" podStartSLOduration=8.816755362 podStartE2EDuration="38.868225338s" totalImagesPullingTime="30.051469976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:14.866717167 +0000 UTC m=+39.902526873" watchObservedRunningTime="2026-07-09 09:34:14.868225338 +0000 UTC m=+39.904035043"
Jul 09 09:34:14.997439 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:14.997390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:15.097342 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.097261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:15.335694 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.335665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:15.394168 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.394105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:15.536475 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.536448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:15.536591 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:15.536536    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:15.536656 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.536620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:15.536707 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:15.536687    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:15.536797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:15.536779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:15.536874 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:15.536857    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:16.155835 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:16.155801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:16.430797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:16.430724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:16.526394 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:16.526371    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:16.636755 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:16.636724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:16.756062 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:16.756035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:17.538747 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:17.538722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:17.539100 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:17.538819    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlkst" podUID="fb2bea31-b084-4006-9ea1-02113a6eeb22"
Jul 09 09:34:17.539100 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:17.538858    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:17.539100 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:17.538930    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws58m" podUID="53ce9f2d-7e9f-4933-aa70-236be2d8b270"
Jul 09 09:34:17.539100 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:17.538970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:17.539100 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:17.539055    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29jzx" podUID="021994c4-fc8b-4b9c-b154-569016c95dfe"
Jul 09 09:34:18.259519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.259451    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-211.ec2.internal" event="NodeReady"
Jul 09 09:34:18.259653 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.259539    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 09:34:18.292981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.292955    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9547fb7d9-5vjpd"]
Jul 09 09:34:18.309416 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.309392    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jbr8d"]
Jul 09 09:34:18.309510 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.309501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.312001 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.311979    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-45tlw\""
Jul 09 09:34:18.312110 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.311985    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 09:34:18.312110 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.312044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 09:34:18.312110 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.312056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 09:34:18.320884 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.320857    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 09:34:18.323210 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.323171    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9547fb7d9-5vjpd"]
Jul 09 09:34:18.323282 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.323228    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-glzrj"]
Jul 09 09:34:18.323330 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.323298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.325925 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.325912    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mlrpb\""
Jul 09 09:34:18.326223 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.326210    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 09:34:18.326313 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.326300    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 09:34:18.343390 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.343373    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbr8d"]
Jul 09 09:34:18.343468 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.343406    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glzrj"]
Jul 09 09:34:18.343468 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.343442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.345791 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.345766    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sxdgx\""
Jul 09 09:34:18.345886 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.345838    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 09:34:18.345886 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.345840    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 09:34:18.345992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.345895    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 09:34:18.345992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.345900    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 09:34:18.409315 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.409290    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wsv"]
Jul 09 09:34:18.429526 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.429502    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wsv"]
Jul 09 09:34:18.429675 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.429609    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.432571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.432545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 09:34:18.432571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.432557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ckfzz\""
Jul 09 09:34:18.432909 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.432892    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 09:34:18.432992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.432960    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 09:34:18.481750 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.481722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:18.487834 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66291734-ad8e-470c-9aa9-1069950ad995-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.487899 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-image-registry-private-configuration\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.487899 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-registry-certificates\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.487899 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-installation-pull-secrets\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.487994 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-config-volume\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.488058 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.487997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66291734-ad8e-470c-9aa9-1069950ad995-crio-socket\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.488058 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-trusted-ca\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.488058 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbn6\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-kube-api-access-wpbn6\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.488150 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxs9b\" (UniqueName: \"kubernetes.io/projected/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-kube-api-access-cxs9b\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.488150 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6nm\" (UniqueName: \"kubernetes.io/projected/66291734-ad8e-470c-9aa9-1069950ad995-kube-api-access-5r6nm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.488150 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66291734-ad8e-470c-9aa9-1069950ad995-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.488150 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-metrics-tls\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.488295 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-bound-sa-token\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.488295 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-tmp-dir\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.488295 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-registry-tls\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.488390 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66291734-ad8e-470c-9aa9-1069950ad995-data-volume\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.488390 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.488322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f18fc5-6c98-407f-a197-46ad302411d2-ca-trust-extracted\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.545672 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.545605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:18.588746 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6nm\" (UniqueName: \"kubernetes.io/projected/66291734-ad8e-470c-9aa9-1069950ad995-kube-api-access-5r6nm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.588823 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9g77\" (UniqueName: \"kubernetes.io/projected/66d45501-b2b0-4850-acbe-ba3f3bf2a835-kube-api-access-r9g77\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.588823 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66291734-ad8e-470c-9aa9-1069950ad995-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.588823 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-metrics-tls\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.588952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-bound-sa-token\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.588952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-tmp-dir\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.588952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-registry-tls\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.588952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66291734-ad8e-470c-9aa9-1069950ad995-data-volume\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.588954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f18fc5-6c98-407f-a197-46ad302411d2-ca-trust-extracted\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66291734-ad8e-470c-9aa9-1069950ad995-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-image-registry-private-configuration\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-registry-certificates\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-installation-pull-secrets\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.589215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-config-volume\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.589550 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-tmp-dir\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.589720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66291734-ad8e-470c-9aa9-1069950ad995-crio-socket\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.589899 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-trusted-ca\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.589996 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbn6\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-kube-api-access-wpbn6\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.590366 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589717    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66291734-ad8e-470c-9aa9-1069950ad995-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.594846 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.590454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-registry-certificates\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.594952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.590853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66291734-ad8e-470c-9aa9-1069950ad995-data-volume\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.594952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.590959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-config-volume\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.594952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.589897    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f18fc5-6c98-407f-a197-46ad302411d2-ca-trust-extracted\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.594952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.591818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d45501-b2b0-4850-acbe-ba3f3bf2a835-cert\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.594952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.594945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxs9b\" (UniqueName: \"kubernetes.io/projected/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-kube-api-access-cxs9b\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.595218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.593990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66291734-ad8e-470c-9aa9-1069950ad995-crio-socket\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.595218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.593755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f18fc5-6c98-407f-a197-46ad302411d2-trusted-ca\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.595586 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.595508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-metrics-tls\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.595586 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.595521    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66291734-ad8e-470c-9aa9-1069950ad995-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.595720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.595561    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-image-registry-private-configuration\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.596323 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.596299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-registry-tls\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.597162 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.597138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f18fc5-6c98-407f-a197-46ad302411d2-installation-pull-secrets\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.597615 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.597589    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6nm\" (UniqueName: \"kubernetes.io/projected/66291734-ad8e-470c-9aa9-1069950ad995-kube-api-access-5r6nm\") pod \"insights-runtime-extractor-glzrj\" (UID: \"66291734-ad8e-470c-9aa9-1069950ad995\") " pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.597856 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.597838    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbn6\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-kube-api-access-wpbn6\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.598005 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.597989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f18fc5-6c98-407f-a197-46ad302411d2-bound-sa-token\") pod \"image-registry-9547fb7d9-5vjpd\" (UID: \"10f18fc5-6c98-407f-a197-46ad302411d2\") " pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.601843 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.601819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxs9b\" (UniqueName: \"kubernetes.io/projected/c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329-kube-api-access-cxs9b\") pod \"dns-default-jbr8d\" (UID: \"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329\") " pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.619733 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.619716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:18.631864 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.631847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:18.650992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.650971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glzrj"
Jul 09 09:34:18.696027 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.696003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d45501-b2b0-4850-acbe-ba3f3bf2a835-cert\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.696154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.696047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9g77\" (UniqueName: \"kubernetes.io/projected/66d45501-b2b0-4850-acbe-ba3f3bf2a835-kube-api-access-r9g77\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.698680 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.698656    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d45501-b2b0-4850-acbe-ba3f3bf2a835-cert\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.709266 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.709201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9g77\" (UniqueName: \"kubernetes.io/projected/66d45501-b2b0-4850-acbe-ba3f3bf2a835-kube-api-access-r9g77\") pod \"ingress-canary-q8wsv\" (UID: \"66d45501-b2b0-4850-acbe-ba3f3bf2a835\") " pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.738071 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.737993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8wsv"
Jul 09 09:34:18.870760 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.870528    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbr8d"]
Jul 09 09:34:18.876148 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.876125    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9547fb7d9-5vjpd"]
Jul 09 09:34:18.876548 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:18.876523    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84ae4b3_3ecf_4ab6_ac3c_348fb7b12329.slice/crio-209b64a7dc2954e60c41374453143e3dbdabb2a139d3185b07bff47f7c4217e8 WatchSource:0}: Error finding container 209b64a7dc2954e60c41374453143e3dbdabb2a139d3185b07bff47f7c4217e8: Status 404 returned error can't find the container with id 209b64a7dc2954e60c41374453143e3dbdabb2a139d3185b07bff47f7c4217e8
Jul 09 09:34:18.878301 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:18.878275    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f18fc5_6c98_407f_a197_46ad302411d2.slice/crio-e9374512617000ad7e5a0b2b95a7458fbcf13e41cfb16eb78f843513fd438c97 WatchSource:0}: Error finding container e9374512617000ad7e5a0b2b95a7458fbcf13e41cfb16eb78f843513fd438c97: Status 404 returned error can't find the container with id e9374512617000ad7e5a0b2b95a7458fbcf13e41cfb16eb78f843513fd438c97
Jul 09 09:34:18.879220 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.879174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glzrj"]
Jul 09 09:34:18.883813 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:18.883793    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66291734_ad8e_470c_9aa9_1069950ad995.slice/crio-2ed39c809d04eb7f9a01dcc314c72e6407a1606027305941c906126f6b1bb25a WatchSource:0}: Error finding container 2ed39c809d04eb7f9a01dcc314c72e6407a1606027305941c906126f6b1bb25a: Status 404 returned error can't find the container with id 2ed39c809d04eb7f9a01dcc314c72e6407a1606027305941c906126f6b1bb25a
Jul 09 09:34:18.899895 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.899870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:18.916859 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:18.916793    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wsv"]
Jul 09 09:34:18.921975 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:18.921951    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d45501_b2b0_4850_acbe_ba3f3bf2a835.slice/crio-ce394ec65b30441cde40d9a873cd8f9678b19b18b100776681d7049b2eabefa3 WatchSource:0}: Error finding container ce394ec65b30441cde40d9a873cd8f9678b19b18b100776681d7049b2eabefa3: Status 404 returned error can't find the container with id ce394ec65b30441cde40d9a873cd8f9678b19b18b100776681d7049b2eabefa3
Jul 09 09:34:19.535857 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.535818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:19.536306 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.536284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:19.536770 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.536712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:19.539528 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.539443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 09:34:19.539964 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.539695    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 09:34:19.540892 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.540331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btgg6\""
Jul 09 09:34:19.540892 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.540546    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 09:34:19.540892 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.540737    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 09:34:19.540892 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.540873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sb7qz\""
Jul 09 09:34:19.653723 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.653683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:19.747171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.747107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:19.852565 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.852028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd" event={"ID":"10f18fc5-6c98-407f-a197-46ad302411d2","Type":"ContainerStarted","Data":"9ca334eeddc4e00362f1586a6099c43ce0e04b50ac9eb5451b33e5154fec7bec"}
Jul 09 09:34:19.852565 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.852086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd" event={"ID":"10f18fc5-6c98-407f-a197-46ad302411d2","Type":"ContainerStarted","Data":"e9374512617000ad7e5a0b2b95a7458fbcf13e41cfb16eb78f843513fd438c97"}
Jul 09 09:34:19.852565 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.852288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:19.855353 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.855318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8wsv" event={"ID":"66d45501-b2b0-4850-acbe-ba3f3bf2a835","Type":"ContainerStarted","Data":"ce394ec65b30441cde40d9a873cd8f9678b19b18b100776681d7049b2eabefa3"}
Jul 09 09:34:19.858828 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.858391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbr8d" event={"ID":"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329","Type":"ContainerStarted","Data":"209b64a7dc2954e60c41374453143e3dbdabb2a139d3185b07bff47f7c4217e8"}
Jul 09 09:34:19.860826 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.860555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glzrj" event={"ID":"66291734-ad8e-470c-9aa9-1069950ad995","Type":"ContainerStarted","Data":"2ed39c809d04eb7f9a01dcc314c72e6407a1606027305941c906126f6b1bb25a"}
Jul 09 09:34:19.879422 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.878759    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd" podStartSLOduration=7.878740781 podStartE2EDuration="7.878740781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:19.878127669 +0000 UTC m=+44.913937375" watchObservedRunningTime="2026-07-09 09:34:19.878740781 +0000 UTC m=+44.914550486"
Jul 09 09:34:19.938047 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:19.938018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:25.878310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.878070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbr8d" event={"ID":"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329","Type":"ContainerStarted","Data":"390f52d16586724e2c6858e9a17a83476cc299180b8e86486fd07ef47bfc1652"}
Jul 09 09:34:25.878933 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.878350    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:25.878933 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.878380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbr8d" event={"ID":"c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329","Type":"ContainerStarted","Data":"a1612a187acd7bcaa8da8ccc8de9abbfa33322a248921735b07acb430095ddc6"}
Jul 09 09:34:25.879758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.879727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glzrj" event={"ID":"66291734-ad8e-470c-9aa9-1069950ad995","Type":"ContainerStarted","Data":"835f1d1c8b7de4b64eb6282b95b842a33b7eeee476849d94aeafdbc4ccd784b0"}
Jul 09 09:34:25.881053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.881029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8wsv" event={"ID":"66d45501-b2b0-4850-acbe-ba3f3bf2a835","Type":"ContainerStarted","Data":"e1cde901464c78b1d020a89e5d6f93432a75aaa6dddba6bb18de03f18df0b3b0"}
Jul 09 09:34:25.936219 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.936150    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jbr8d" podStartSLOduration=2.028536879 podStartE2EDuration="7.936133605s" totalImagesPullingTime="5.907596726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:25.903747229 +0000 UTC m=+50.939556935" watchObservedRunningTime="2026-07-09 09:34:25.936133605 +0000 UTC m=+50.971943292"
Jul 09 09:34:25.936458 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:25.936435    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q8wsv" podStartSLOduration=2.066407652 podStartE2EDuration="7.936426714s" totalImagesPullingTime="5.870019062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:25.936291273 +0000 UTC m=+50.972100961" watchObservedRunningTime="2026-07-09 09:34:25.936426714 +0000 UTC m=+50.972236458"
Jul 09 09:34:26.566905 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:26.566864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:26.887445 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:26.887363    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glzrj" event={"ID":"66291734-ad8e-470c-9aa9-1069950ad995","Type":"ContainerStarted","Data":"9c83aa27a32417772b8fd6cc306b0231387e9c14e1c32fd5e379dbfe6e363c2d"}
Jul 09 09:34:28.401623 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.401586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:28.501349 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.501311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:28.562014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.561976    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:28.713440 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.713408    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:28.823655 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.823625    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:28.919685 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:28.919527    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.048545 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.048475    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.081381 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.081361    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"]
Jul 09 09:34:29.088464 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.088449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.091125 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.091106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 09:34:29.091711 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.091604    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 09:34:29.091788 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.091770    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 09:34:29.092677 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.092660    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 09:34:29.092677 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.092666    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mgkrj\""
Jul 09 09:34:29.092677 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.092666    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 09:34:29.096657 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.096640    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"]
Jul 09 09:34:29.126144 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.126118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.130732 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.130711    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m7664"]
Jul 09 09:34:29.154099 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.154079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.156759 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.156741    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jsrjg\""
Jul 09 09:34:29.157052 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.157037    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 09:34:29.157139 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.157066    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 09:34:29.157395 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.157375    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 09:34:29.181398 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-wtmp\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181483 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.181483 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181558 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181558 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.181558 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngr9w\" (UniqueName: \"kubernetes.io/projected/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-kube-api-access-ngr9w\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-textfile\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-root\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-sys\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-metrics-client-ca\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.181758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.181750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648r8\" (UniqueName: \"kubernetes.io/projected/3b227347-b1fd-456a-aa5f-5ad529d0cb35-kube-api-access-648r8\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.282911 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.282884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-metrics-client-ca\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.282916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648r8\" (UniqueName: \"kubernetes.io/projected/3b227347-b1fd-456a-aa5f-5ad529d0cb35-kube-api-access-648r8\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.282934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-wtmp\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.282975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.283020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.282995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngr9w\" (UniqueName: \"kubernetes.io/projected/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-kube-api-access-ngr9w\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-textfile\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-wtmp\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-root\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-sys\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:29.283349    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:29.283418    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls podName:3b227347-b1fd-456a-aa5f-5ad529d0cb35 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:29.783397755 +0000 UTC m=+54.819207454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls") pod "node-exporter-m7664" (UID: "3b227347-b1fd-456a-aa5f-5ad529d0cb35") : secret "node-exporter-tls" not found
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283447    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-root\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b227347-b1fd-456a-aa5f-5ad529d0cb35-sys\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-metrics-client-ca\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-textfile\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.283912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283806    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.283912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.283855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.286928 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.286908    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.287163 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.287141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.287220 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.287144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.295605 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.295584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-648r8\" (UniqueName: \"kubernetes.io/projected/3b227347-b1fd-456a-aa5f-5ad529d0cb35-kube-api-access-648r8\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.296171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.296154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngr9w\" (UniqueName: \"kubernetes.io/projected/c2422981-b620-4ecc-b9ae-9f4fa0ef5f97-kube-api-access-ngr9w\") pod \"openshift-state-metrics-54cf96dfcf-mhwz8\" (UID: \"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.310205 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.310139    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.399371 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.399345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"
Jul 09 09:34:29.407818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.407677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.509862 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.509826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.530481 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.530453    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8"]
Jul 09 09:34:29.534066 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:29.534043    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2422981_b620_4ecc_b9ae_9f4fa0ef5f97.slice/crio-c4b7f5bfa1ed16f5556443b5f4a191d6095db7a1316564c8bfbec6ee7a42002e WatchSource:0}: Error finding container c4b7f5bfa1ed16f5556443b5f4a191d6095db7a1316564c8bfbec6ee7a42002e: Status 404 returned error can't find the container with id c4b7f5bfa1ed16f5556443b5f4a191d6095db7a1316564c8bfbec6ee7a42002e
Jul 09 09:34:29.609248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.609086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.710467 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.710432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.770938 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.770909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.788695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.788672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.791904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.791888    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b227347-b1fd-456a-aa5f-5ad529d0cb35-node-exporter-tls\") pod \"node-exporter-m7664\" (UID: \"3b227347-b1fd-456a-aa5f-5ad529d0cb35\") " pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:29.870842 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.870729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:29.900392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.900359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glzrj" event={"ID":"66291734-ad8e-470c-9aa9-1069950ad995","Type":"ContainerStarted","Data":"834058cb25b2b698b8fa1472febf50f8b6513292b6e40b66c28bec63b49d5f45"}
Jul 09 09:34:29.902147 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.902125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8" event={"ID":"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97","Type":"ContainerStarted","Data":"093b0394f85644aaf7473113cc59d85a62f5faf6829ccb36be9ab69636cd1039"}
Jul 09 09:34:29.902255 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.902164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8" event={"ID":"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97","Type":"ContainerStarted","Data":"cd690b3ed4f2474fe54c93271ccf0d59ba1b451598f9267e60a1e02b4892e091"}
Jul 09 09:34:29.902255 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.902198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8" event={"ID":"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97","Type":"ContainerStarted","Data":"c4b7f5bfa1ed16f5556443b5f4a191d6095db7a1316564c8bfbec6ee7a42002e"}
Jul 09 09:34:29.926241 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:29.926174    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-glzrj" podStartSLOduration=1.982232679 podStartE2EDuration="11.926159717s" totalImagesPullingTime="9.943927038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:29.925519067 +0000 UTC m=+54.961328772" watchObservedRunningTime="2026-07-09 09:34:29.926159717 +0000 UTC m=+54.961969420"
Jul 09 09:34:30.063020 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.062988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7664"
Jul 09 09:34:30.071936 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:30.071910    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b227347_b1fd_456a_aa5f_5ad529d0cb35.slice/crio-be7e4399bee6c75e133e5a5656e5d12c307cbbb8222d04e13d630010b8991722 WatchSource:0}: Error finding container be7e4399bee6c75e133e5a5656e5d12c307cbbb8222d04e13d630010b8991722: Status 404 returned error can't find the container with id be7e4399bee6c75e133e5a5656e5d12c307cbbb8222d04e13d630010b8991722
Jul 09 09:34:30.089606 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.089581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.188912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.188875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.216589 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.216556    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:34:30.219050 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.219029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.222500 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.222479    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 09:34:30.222636 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.222523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t74jt\""
Jul 09 09:34:30.222767 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.222752    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 09:34:30.223485 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.223467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 09:34:30.223577 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.223469    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 09:34:30.225192 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.225158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 09:34:30.225736 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.225717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 09:34:30.226169 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.226030    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 09:34:30.226277 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.226170    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 09:34:30.226535 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.226514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 09:34:30.253997 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.253964    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:34:30.287786 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.287755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.292903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.292876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.292910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.292944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293156 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293156 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293407 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwmh\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293407 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293407 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.293549 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.293423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.351628 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.351596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.394236 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394236 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394236 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:34:30.394627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle podName:3ca280d6-0370-4693-be2c-f495f57bf07e nodeName:}" failed. No retries permitted until 2026-07-09 09:34:30.894601345 +0000 UTC m=+55.930411027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e") : configmap references non-existent config key: ca-bundle.crt
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwmh\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.394753 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.394680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.395553 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.395222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.396982 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.396933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398275 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398205    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398712 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398690    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398785 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.398952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.398928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.399553 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.399534    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.399902 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.399881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.400321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.400304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.411797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.411764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwmh\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.423453 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.423425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.849318 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.849239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:30.899247 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.899216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.899956 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.899929    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:30.907452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.907423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7664" event={"ID":"3b227347-b1fd-456a-aa5f-5ad529d0cb35","Type":"ContainerStarted","Data":"be7e4399bee6c75e133e5a5656e5d12c307cbbb8222d04e13d630010b8991722"}
Jul 09 09:34:30.909713 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.909690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8" event={"ID":"c2422981-b620-4ecc-b9ae-9f4fa0ef5f97","Type":"ContainerStarted","Data":"a572b9e8fbbf9c5961b07b870a923454bf60843d1c8a14708bb3b033596ba409"}
Jul 09 09:34:30.927215 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.927153    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhwz8" podStartSLOduration=1.0171315 podStartE2EDuration="1.927137001s" totalImagesPullingTime="910.005501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:30.925882745 +0000 UTC m=+55.961692451" watchObservedRunningTime="2026-07-09 09:34:30.927137001 +0000 UTC m=+55.962946711"
Jul 09 09:34:30.931480 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:30.931460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:31.130501 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.130475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:34:31.162162 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.162136    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-94785f998-rpp9k"]
Jul 09 09:34:31.164933 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.164918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.167845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.167672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 09:34:31.167845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.167691    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 09:34:31.167845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.167672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 09:34:31.167845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.167751    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 09:34:31.168101 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.168018    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5jk6jbj3qdhs4\""
Jul 09 09:34:31.168231 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.168214    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lspfq\""
Jul 09 09:34:31.168319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.168280    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 09:34:31.179491 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.179450    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94785f998-rpp9k"]
Jul 09 09:34:31.202227 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202322 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/667749e1-7115-4226-b246-1af1a3fde4e5-metrics-client-ca\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202322 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl88c\" (UniqueName: \"kubernetes.io/projected/667749e1-7115-4226-b246-1af1a3fde4e5-kube-api-access-zl88c\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-grpc-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.202571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.202490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.268258 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.268236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:34:31.270818 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:31.270791    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca280d6_0370_4693_be2c_f495f57bf07e.slice/crio-64aff45f1e06d808f9a52215af190b48c630638c9fb987b2a4f20908bd3f674a WatchSource:0}: Error finding container 64aff45f1e06d808f9a52215af190b48c630638c9fb987b2a4f20908bd3f674a: Status 404 returned error can't find the container with id 64aff45f1e06d808f9a52215af190b48c630638c9fb987b2a4f20908bd3f674a
Jul 09 09:34:31.303470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303575 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303575 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl88c\" (UniqueName: \"kubernetes.io/projected/667749e1-7115-4226-b246-1af1a3fde4e5-kube-api-access-zl88c\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303575 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-grpc-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303776 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303833 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303901 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.303946 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.303929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/667749e1-7115-4226-b246-1af1a3fde4e5-metrics-client-ca\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.304695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.304667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/667749e1-7115-4226-b246-1af1a3fde4e5-metrics-client-ca\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306163 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306140    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306272 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-grpc-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306345 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306391 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-tls\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.306573 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.306554    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/667749e1-7115-4226-b246-1af1a3fde4e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.311425 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.311408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl88c\" (UniqueName: \"kubernetes.io/projected/667749e1-7115-4226-b246-1af1a3fde4e5-kube-api-access-zl88c\") pod \"thanos-querier-94785f998-rpp9k\" (UID: \"667749e1-7115-4226-b246-1af1a3fde4e5\") " pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.490639 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.490614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:31.614475 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.614440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94785f998-rpp9k"]
Jul 09 09:34:31.619326 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:31.619300    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667749e1_7115_4226_b246_1af1a3fde4e5.slice/crio-b43b47b59bef48896978c1b1d55d075359cc38151c4466cd4b7b3ff44a4571c3 WatchSource:0}: Error finding container b43b47b59bef48896978c1b1d55d075359cc38151c4466cd4b7b3ff44a4571c3: Status 404 returned error can't find the container with id b43b47b59bef48896978c1b1d55d075359cc38151c4466cd4b7b3ff44a4571c3
Jul 09 09:34:31.695559 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.695530    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:31.797519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.797442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:31.914365 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.914323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"b43b47b59bef48896978c1b1d55d075359cc38151c4466cd4b7b3ff44a4571c3"}
Jul 09 09:34:31.915842 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.915789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"64aff45f1e06d808f9a52215af190b48c630638c9fb987b2a4f20908bd3f674a"}
Jul 09 09:34:31.917672 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.917635    2576 generic.go:356] "Generic (PLEG): container finished" podID="3b227347-b1fd-456a-aa5f-5ad529d0cb35" containerID="020fade39446c7c167c445e0fc34dc8e265c14b75d7b771cb46171800dc9aeba" exitCode=0
Jul 09 09:34:31.917814 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:31.917712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7664" event={"ID":"3b227347-b1fd-456a-aa5f-5ad529d0cb35","Type":"ContainerDied","Data":"020fade39446c7c167c445e0fc34dc8e265c14b75d7b771cb46171800dc9aeba"}
Jul 09 09:34:32.052024 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.051969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:32.174104 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.174071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:32.273674 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.273640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:32.328789 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.328699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:32.923944 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.923912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7664" event={"ID":"3b227347-b1fd-456a-aa5f-5ad529d0cb35","Type":"ContainerStarted","Data":"5ae08d93c8277a7da22a245975755b0cd542312ed6d4838bc6106d9d2cab570c"}
Jul 09 09:34:32.924439 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.924002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7664" event={"ID":"3b227347-b1fd-456a-aa5f-5ad529d0cb35","Type":"ContainerStarted","Data":"0b10bc855fcd7b291d107e3438ecefdeca6247d99abe42f49e273dcf5edf7490"}
Jul 09 09:34:32.925682 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.925657    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf" exitCode=0
Jul 09 09:34:32.925791 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.925729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"}
Jul 09 09:34:32.950615 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:32.950574    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m7664" podStartSLOduration=2.833720618 podStartE2EDuration="3.950558729s" totalImagesPullingTime="1.047785019s" totalInitContainerRuntime="69.053092ms" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:32.948906623 +0000 UTC m=+57.984716325" watchObservedRunningTime="2026-07-09 09:34:32.950558729 +0000 UTC m=+57.986368434"
Jul 09 09:34:33.506605 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.506565    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-758f5fb9d-bz8j2"]
Jul 09 09:34:33.508818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.508798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.511831 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.511810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-86gucb3mog6sd\""
Jul 09 09:34:33.512673 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.512649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 09:34:33.512673 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.512660    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 09:34:33.512827 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.512674    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 09:34:33.513760 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.513737    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-79dzj\""
Jul 09 09:34:33.524601 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.524580    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 09:34:33.526313 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.526267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-758f5fb9d-bz8j2"]
Jul 09 09:34:33.624049 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3b2dc96-3727-45dd-aeec-795763c790b9-audit-log\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-client-certs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624207 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-client-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624257 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624309 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-metrics-server-audit-profiles\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624345 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpcs\" (UniqueName: \"kubernetes.io/projected/c3b2dc96-3727-45dd-aeec-795763c790b9-kube-api-access-dvpcs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.624378 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.624347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-tls\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.724763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-client-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.724904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.724904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-metrics-server-audit-profiles\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.724904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpcs\" (UniqueName: \"kubernetes.io/projected/c3b2dc96-3727-45dd-aeec-795763c790b9-kube-api-access-dvpcs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.724904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-tls\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.725121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3b2dc96-3727-45dd-aeec-795763c790b9-audit-log\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.725121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.724994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-client-certs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.725949 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.725618    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3b2dc96-3727-45dd-aeec-795763c790b9-audit-log\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.726147 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.726097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.726461 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.726438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3b2dc96-3727-45dd-aeec-795763c790b9-metrics-server-audit-profiles\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.727433 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.727411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-client-certs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.727876 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.727854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-client-ca-bundle\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.728113 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.728098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3b2dc96-3727-45dd-aeec-795763c790b9-secret-metrics-server-tls\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.734142 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.734117    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpcs\" (UniqueName: \"kubernetes.io/projected/c3b2dc96-3727-45dd-aeec-795763c790b9-kube-api-access-dvpcs\") pod \"metrics-server-758f5fb9d-bz8j2\" (UID: \"c3b2dc96-3727-45dd-aeec-795763c790b9\") " pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.820035 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.819920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:33.827498 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.827471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:33.915619 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.915586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:33.936417 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.933075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"9f4cb3d013ac941d38e6d6a4eae51be0363c857120efa46df30c5e6440ca00ab"}
Jul 09 09:34:33.936417 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.933239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"722423a1f271213b0633269501d115f5628301cf20429ba5c774c7cd6ba0cf0c"}
Jul 09 09:34:33.936417 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.933307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"996252aeaabbdf2f3e8f760e840e32ce60e3845bbe5989151f06256b400960dd"}
Jul 09 09:34:33.969279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:33.969247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-758f5fb9d-bz8j2"]
Jul 09 09:34:33.972283 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:33.972256    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b2dc96_3727_45dd_aeec_795763c790b9.slice/crio-76c39083cc790e059a7e179aae2d6c4fc58cc241b4e3f236c8abdaa93c4cbaa8 WatchSource:0}: Error finding container 76c39083cc790e059a7e179aae2d6c4fc58cc241b4e3f236c8abdaa93c4cbaa8: Status 404 returned error can't find the container with id 76c39083cc790e059a7e179aae2d6c4fc58cc241b4e3f236c8abdaa93c4cbaa8
Jul 09 09:34:34.000101 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:34.000066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:34.936854 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:34.936824    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2" event={"ID":"c3b2dc96-3727-45dd-aeec-795763c790b9","Type":"ContainerStarted","Data":"76c39083cc790e059a7e179aae2d6c4fc58cc241b4e3f236c8abdaa93c4cbaa8"}
Jul 09 09:34:35.409520 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.409480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:35.889703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.889678    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jbr8d"
Jul 09 09:34:35.943708 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.943674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"4d93a6ac9073b9a7dc076acffba49224f76e3e22fdc307395506f6bb0d1c2758"}
Jul 09 09:34:35.944089 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.943916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:35.944270 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.944230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"76013b2140e7ad56d2aa8d03a571c885f9d65a8069cad35e06a1c8c3c37ea2f9"}
Jul 09 09:34:35.944390 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.944298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" event={"ID":"667749e1-7115-4226-b246-1af1a3fde4e5","Type":"ContainerStarted","Data":"603f0ee6b114d1f6c7fc42d3368805ed5e70494143f8ba27446e5836fd70b7a4"}
Jul 09 09:34:35.947662 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.947637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"}
Jul 09 09:34:35.947766 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.947706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"}
Jul 09 09:34:35.947766 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:35.947751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"}
Jul 09 09:34:36.815118 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.815092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdmzb"
Jul 09 09:34:36.843258 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.843229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:36.849320 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.849279    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k" podStartSLOduration=3.06265501 podStartE2EDuration="5.84926372s" totalImagesPullingTime="2.78660871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:35.981848253 +0000 UTC m=+61.017657957" watchObservedRunningTime="2026-07-09 09:34:36.84926372 +0000 UTC m=+61.885073424"
Jul 09 09:34:36.938308 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.938278    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:36.954326 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.954293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"}
Jul 09 09:34:36.954723 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.954373    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"}
Jul 09 09:34:36.954723 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.954418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerStarted","Data":"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"}
Jul 09 09:34:36.955706 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.955678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2" event={"ID":"c3b2dc96-3727-45dd-aeec-795763c790b9","Type":"ContainerStarted","Data":"e977310777086c31b5e13b3680fa0a4d995d8a6640c434063e0ec193673e09e3"}
Jul 09 09:34:36.985220 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:36.985155    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.569201296 podStartE2EDuration="6.985141875s" totalImagesPullingTime="3.395181397s" totalInitContainerRuntime="20.759182ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:36.983792069 +0000 UTC m=+62.019601806" watchObservedRunningTime="2026-07-09 09:34:36.985141875 +0000 UTC m=+62.020951580"
Jul 09 09:34:37.000250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:37.000196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:37.006638 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:37.006596    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2" podStartSLOduration=1.759645152 podStartE2EDuration="4.006579881s" totalImagesPullingTime="2.246934729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:37.006277369 +0000 UTC m=+62.042087078" watchObservedRunningTime="2026-07-09 09:34:37.006579881 +0000 UTC m=+62.042389587"
Jul 09 09:34:37.076510 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:37.076439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:38.624144 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:38.624111    2576 patch_prober.go:28] interesting pod/image-registry-9547fb7d9-5vjpd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 09:34:38.624554 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:38.624162    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd" podUID="10f18fc5-6c98-407f-a197-46ad302411d2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:34:40.355488 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:40.355451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:40.458245 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:40.458214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:40.868210 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:40.868167    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9547fb7d9-5vjpd"
Jul 09 09:34:41.024577 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.024540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:41.123494 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.123413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:41.223871 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.223842    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:41.282983 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.282955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:41.389872 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.389817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:41.389872 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.389854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:41.393137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.393119    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 09:34:41.394068 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.394055    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 09:34:41.403261 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.403241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ce9f2d-7e9f-4933-aa70-236be2d8b270-metrics-certs\") pod \"network-metrics-daemon-ws58m\" (UID: \"53ce9f2d-7e9f-4933-aa70-236be2d8b270\") " pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:41.403366 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.403283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/021994c4-fc8b-4b9c-b154-569016c95dfe-original-pull-secret\") pod \"global-pull-secret-syncer-29jzx\" (UID: \"021994c4-fc8b-4b9c-b154-569016c95dfe\") " pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:41.464195 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.464164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btgg6\""
Jul 09 09:34:41.471426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.471408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29jzx"
Jul 09 09:34:41.472468 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.472455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws58m"
Jul 09 09:34:41.491226 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.491196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:41.494273 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.494253    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 09:34:41.504403 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.504382    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 09:34:41.515377 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.515354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgf4t\" (UniqueName: \"kubernetes.io/projected/fb2bea31-b084-4006-9ea1-02113a6eeb22-kube-api-access-jgf4t\") pod \"network-check-target-nlkst\" (UID: \"fb2bea31-b084-4006-9ea1-02113a6eeb22\") " pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:41.633197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.633149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29jzx"]
Jul 09 09:34:41.636550 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:41.636522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021994c4_fc8b_4b9c_b154_569016c95dfe.slice/crio-eda297f9de5b3b8baca0162091b38da0fd4326f98244eb860ef403b754599462 WatchSource:0}: Error finding container eda297f9de5b3b8baca0162091b38da0fd4326f98244eb860ef403b754599462: Status 404 returned error can't find the container with id eda297f9de5b3b8baca0162091b38da0fd4326f98244eb860ef403b754599462
Jul 09 09:34:41.650820 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.650794    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ws58m"]
Jul 09 09:34:41.653564 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:41.653539    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ce9f2d_7e9f_4933_aa70_236be2d8b270.slice/crio-b268a4f80fb67eabdde7a03f4ba74d06c0aa31be1b05081775560cd8e8bcb3f3 WatchSource:0}: Error finding container b268a4f80fb67eabdde7a03f4ba74d06c0aa31be1b05081775560cd8e8bcb3f3: Status 404 returned error can't find the container with id b268a4f80fb67eabdde7a03f4ba74d06c0aa31be1b05081775560cd8e8bcb3f3
Jul 09 09:34:41.756994 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.756957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sb7qz\""
Jul 09 09:34:41.764438 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.764416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:41.889858 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.889824    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlkst"]
Jul 09 09:34:41.892800 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:41.892776    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2bea31_b084_4006_9ea1_02113a6eeb22.slice/crio-79569ec03b99ec6335760de79af7b0c03d59ef004b719125699fa720600a0373 WatchSource:0}: Error finding container 79569ec03b99ec6335760de79af7b0c03d59ef004b719125699fa720600a0373: Status 404 returned error can't find the container with id 79569ec03b99ec6335760de79af7b0c03d59ef004b719125699fa720600a0373
Jul 09 09:34:41.962011 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.961981    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-94785f998-rpp9k"
Jul 09 09:34:41.974668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.974634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlkst" event={"ID":"fb2bea31-b084-4006-9ea1-02113a6eeb22","Type":"ContainerStarted","Data":"79569ec03b99ec6335760de79af7b0c03d59ef004b719125699fa720600a0373"}
Jul 09 09:34:41.975782 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.975752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29jzx" event={"ID":"021994c4-fc8b-4b9c-b154-569016c95dfe","Type":"ContainerStarted","Data":"eda297f9de5b3b8baca0162091b38da0fd4326f98244eb860ef403b754599462"}
Jul 09 09:34:41.976813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:41.976789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws58m" event={"ID":"53ce9f2d-7e9f-4933-aa70-236be2d8b270","Type":"ContainerStarted","Data":"b268a4f80fb67eabdde7a03f4ba74d06c0aa31be1b05081775560cd8e8bcb3f3"}
Jul 09 09:34:42.702308 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.702272    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:34:42.715737 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.715714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.719150 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.719129    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 09:34:42.720493 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:34:42.720493 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720418    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 09:34:42.720655 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720558    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5sxl6\""
Jul 09 09:34:42.720655 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720606    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 09:34:42.720838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720817    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 09:34:42.720972 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.720959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 09:34:42.722851 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.721131    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 09:34:42.722851 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.722173    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 09:34:42.725871 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.725851    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 09:34:42.802167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802283 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802283 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4cq\" (UniqueName: \"kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802504 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.802554 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.802520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903591 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4cq\" (UniqueName: \"kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.903765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.903760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.904473 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.904418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.904580 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.904500    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.904720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.904693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.905043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.904995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.906572 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.906550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.907135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.907100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:42.919965 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:42.919940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4cq\" (UniqueName: \"kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq\") pod \"console-6b6b9684f6-mx7q9\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") " pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:43.030609 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.030525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:43.237528 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.237451    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:34:43.244245 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:34:43.244221    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b98831_85e9_4d85_9736_cfa863a9cabc.slice/crio-a0143036111ec933e6f7ddc73c698a27f7dd06b379181df1f89c8068f7b6a2a2 WatchSource:0}: Error finding container a0143036111ec933e6f7ddc73c698a27f7dd06b379181df1f89c8068f7b6a2a2: Status 404 returned error can't find the container with id a0143036111ec933e6f7ddc73c698a27f7dd06b379181df1f89c8068f7b6a2a2
Jul 09 09:34:43.505269 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.505171    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:43.991882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.991841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws58m" event={"ID":"53ce9f2d-7e9f-4933-aa70-236be2d8b270","Type":"ContainerStarted","Data":"fd75dd7517e3134cb790ce43c0eb4fa85cd5f1f7537e360798e31e5890a35d4d"}
Jul 09 09:34:43.992374 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.991913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws58m" event={"ID":"53ce9f2d-7e9f-4933-aa70-236be2d8b270","Type":"ContainerStarted","Data":"4899e787d60020fd50a635ed20766eaf6d11a0811616f10324560cf374a443a2"}
Jul 09 09:34:43.993438 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:43.993379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6b9684f6-mx7q9" event={"ID":"b7b98831-85e9-4d85-9736-cfa863a9cabc","Type":"ContainerStarted","Data":"a0143036111ec933e6f7ddc73c698a27f7dd06b379181df1f89c8068f7b6a2a2"}
Jul 09 09:34:44.015172 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:44.014967    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ws58m" podStartSLOduration=66.589049077 podStartE2EDuration="1m8.014948966s" totalImagesPullingTime="1.425899889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:44.013708268 +0000 UTC m=+69.049517974" watchObservedRunningTime="2026-07-09 09:34:44.014948966 +0000 UTC m=+69.050758671"
Jul 09 09:34:44.142976 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:44.142909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:44.210747 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:44.210624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:44.610233 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:44.610121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:44.709900 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:44.709858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:45.039912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:45.039875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:45.136112 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:45.136076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:46.285964 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:46.285930    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:46.420993 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:46.420952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:46.750674 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:46.750623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:46.809656 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:46.809623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:46.904998 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:46.904956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:48.014329 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.014295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6b9684f6-mx7q9" event={"ID":"b7b98831-85e9-4d85-9736-cfa863a9cabc","Type":"ContainerStarted","Data":"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"}
Jul 09 09:34:48.015913 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.015876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlkst" event={"ID":"fb2bea31-b084-4006-9ea1-02113a6eeb22","Type":"ContainerStarted","Data":"ea583a2070ec746a2c85cddc989b31a5380c1d3e9da22cc96d4aa5cf117c7dd6"}
Jul 09 09:34:48.016119 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.015937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:34:48.017394 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.017371    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29jzx" event={"ID":"021994c4-fc8b-4b9c-b154-569016c95dfe","Type":"ContainerStarted","Data":"53a951ac7ee174afa2d5e7ad4236ecb66cf558af7d0f682598ca58e6b205e192"}
Jul 09 09:34:48.043210 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.043123    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6b9684f6-mx7q9" podStartSLOduration=1.628382218 podStartE2EDuration="6.043100131s" totalImagesPullingTime="4.414717913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:48.041490092 +0000 UTC m=+73.077299798" watchObservedRunningTime="2026-07-09 09:34:48.043100131 +0000 UTC m=+73.078909837"
Jul 09 09:34:48.067321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.067260    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-29jzx" podStartSLOduration=66.038953582 podStartE2EDuration="1m12.067238716s" totalImagesPullingTime="6.028285134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:48.064810458 +0000 UTC m=+73.100620165" watchObservedRunningTime="2026-07-09 09:34:48.067238716 +0000 UTC m=+73.103048429"
Jul 09 09:34:48.112421 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.112358    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nlkst" podStartSLOduration=66.35213395 podStartE2EDuration="1m12.112337801s" totalImagesPullingTime="5.760203851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:48.111996886 +0000 UTC m=+73.147806591" watchObservedRunningTime="2026-07-09 09:34:48.112337801 +0000 UTC m=+73.148147507"
Jul 09 09:34:48.765052 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.765022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:48.863364 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:48.863330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:52.129952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:52.129916    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:52.191761 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:52.191731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:53.031310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.031278    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:53.031508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.031374    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:53.036260 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.036239    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:53.076160 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.076133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:53.136088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.136063    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:53.745314 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.745284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:53.819996 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.819972    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:53.820171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.820034    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:34:53.821077 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:53.821052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:54.041727 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:54.041653    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:34:54.158750 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:54.158720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:56.102919 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:56.102890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:56.305607 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:56.305575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:56.402529 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:56.402464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:56.611463 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:56.611431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:34:56.712565 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:34:56.712540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:01.648401 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:01.648363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:01.749315 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:01.749280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:13.825361 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:13.825333    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:35:13.829328 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:13.829292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-758f5fb9d-bz8j2"
Jul 09 09:35:16.617007 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:16.616946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:16.719345 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:16.719304    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:17.734805 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:17.734769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:17.791566 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:17.791532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:19.023224 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:19.023172    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nlkst"
Jul 09 09:35:23.176795 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:23.176760    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:23.237670 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:23.237642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:35.168314 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:35.168282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:35.227429 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:35.227323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:48.309968 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:48.309931    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:48.761402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:48.761364    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:49.376380 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376346    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:49.376783 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376649    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="alertmanager" containerID="cri-o://fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8" gracePeriod=120
Jul 09 09:35:49.376783 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376715    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-metric" containerID="cri-o://a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d" gracePeriod=120
Jul 09 09:35:49.376783 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376724    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-web" containerID="cri-o://e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf" gracePeriod=120
Jul 09 09:35:49.376783 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376761    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy" containerID="cri-o://927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e" gracePeriod=120
Jul 09 09:35:49.376984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376732    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="config-reloader" containerID="cri-o://81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf" gracePeriod=120
Jul 09 09:35:49.376984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:49.376810    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="prom-label-proxy" containerID="cri-o://26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6" gracePeriod=120
Jul 09 09:35:50.246084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246052    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6" exitCode=0
Jul 09 09:35:50.246084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246078    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e" exitCode=0
Jul 09 09:35:50.246084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246084    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf" exitCode=0
Jul 09 09:35:50.246084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246089    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8" exitCode=0
Jul 09 09:35:50.246370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"}
Jul 09 09:35:50.246370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246216    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"}
Jul 09 09:35:50.246370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"}
Jul 09 09:35:50.246370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.246270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"}
Jul 09 09:35:50.619894 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.619871    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:50.737952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.737916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.737963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.737990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwmh\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738096    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738130    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738260    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738349    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738381    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738415    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") pod \"3ca280d6-0370-4693-be2c-f495f57bf07e\" (UID: \"3ca280d6-0370-4693-be2c-f495f57bf07e\") "
Jul 09 09:35:50.738503 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738477    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:35:50.738827 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738671    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-metrics-client-ca\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.738927 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.738904    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:35:50.740433 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.740394    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:35:50.741405 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.741364    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:35:50.741522 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.741404    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.741594 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.741560    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.741706 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.741671    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume" (OuterVolumeSpecName: "config-volume") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.742053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.742025    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.742148 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.742055    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh" (OuterVolumeSpecName: "kube-api-access-xvwmh") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "kube-api-access-xvwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:35:50.742382 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.742362    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.742618 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.742595    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out" (OuterVolumeSpecName: "config-out") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:35:50.745496 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.745372    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.752583 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.752560    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config" (OuterVolumeSpecName: "web-config") pod "3ca280d6-0370-4693-be2c-f495f57bf07e" (UID: "3ca280d6-0370-4693-be2c-f495f57bf07e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:35:50.839036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.838969    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-main-db\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839011    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-web-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839021    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca280d6-0370-4693-be2c-f495f57bf07e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839036 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839032    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839042    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-main-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839051    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839059    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwmh\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-kube-api-access-xvwmh\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839067    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca280d6-0370-4693-be2c-f495f57bf07e-tls-assets\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839075    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-config-volume\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839082    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839090    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca280d6-0370-4693-be2c-f495f57bf07e-config-out\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:50.839259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:50.839101    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ca280d6-0370-4693-be2c-f495f57bf07e-cluster-tls-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:35:51.252982 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.252946    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d" exitCode=0
Jul 09 09:35:51.252982 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.252980    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerID="e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf" exitCode=0
Jul 09 09:35:51.253197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.253019    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"}
Jul 09 09:35:51.253197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.253040    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.253197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.253090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"}
Jul 09 09:35:51.253197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.253120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ca280d6-0370-4693-be2c-f495f57bf07e","Type":"ContainerDied","Data":"64aff45f1e06d808f9a52215af190b48c630638c9fb987b2a4f20908bd3f674a"}
Jul 09 09:35:51.253197 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.253150    2576 scope.go:117] "RemoveContainer" containerID="26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"
Jul 09 09:35:51.261383 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.261368    2576 scope.go:117] "RemoveContainer" containerID="a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"
Jul 09 09:35:51.268017 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.268000    2576 scope.go:117] "RemoveContainer" containerID="927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"
Jul 09 09:35:51.274221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.274201    2576 scope.go:117] "RemoveContainer" containerID="e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"
Jul 09 09:35:51.277972 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.277949    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:51.281305 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.281274    2576 scope.go:117] "RemoveContainer" containerID="81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"
Jul 09 09:35:51.281777 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.281756    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:51.287701 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.287687    2576 scope.go:117] "RemoveContainer" containerID="fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"
Jul 09 09:35:51.293948 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.293934    2576 scope.go:117] "RemoveContainer" containerID="d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"
Jul 09 09:35:51.300080 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300065    2576 scope.go:117] "RemoveContainer" containerID="26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"
Jul 09 09:35:51.300355 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.300335    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6\": container with ID starting with 26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6 not found: ID does not exist" containerID="26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"
Jul 09 09:35:51.300402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300363    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"} err="failed to get container status \"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6\": rpc error: code = NotFound desc = could not find container \"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6\": container with ID starting with 26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6 not found: ID does not exist"
Jul 09 09:35:51.300402 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300391    2576 scope.go:117] "RemoveContainer" containerID="a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"
Jul 09 09:35:51.300603 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.300588    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d\": container with ID starting with a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d not found: ID does not exist" containerID="a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"
Jul 09 09:35:51.300645 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300610    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"} err="failed to get container status \"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d\": rpc error: code = NotFound desc = could not find container \"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d\": container with ID starting with a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d not found: ID does not exist"
Jul 09 09:35:51.300645 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300625    2576 scope.go:117] "RemoveContainer" containerID="927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"
Jul 09 09:35:51.300810 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.300796    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e\": container with ID starting with 927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e not found: ID does not exist" containerID="927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"
Jul 09 09:35:51.300850 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300815    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"} err="failed to get container status \"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e\": rpc error: code = NotFound desc = could not find container \"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e\": container with ID starting with 927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e not found: ID does not exist"
Jul 09 09:35:51.300850 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.300833    2576 scope.go:117] "RemoveContainer" containerID="e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"
Jul 09 09:35:51.301020 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.301000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf\": container with ID starting with e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf not found: ID does not exist" containerID="e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"
Jul 09 09:35:51.301060 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301027    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"} err="failed to get container status \"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf\": rpc error: code = NotFound desc = could not find container \"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf\": container with ID starting with e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf not found: ID does not exist"
Jul 09 09:35:51.301060 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301045    2576 scope.go:117] "RemoveContainer" containerID="81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"
Jul 09 09:35:51.301314 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.301299    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf\": container with ID starting with 81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf not found: ID does not exist" containerID="81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"
Jul 09 09:35:51.301386 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301317    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"} err="failed to get container status \"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf\": rpc error: code = NotFound desc = could not find container \"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf\": container with ID starting with 81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf not found: ID does not exist"
Jul 09 09:35:51.301386 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301329    2576 scope.go:117] "RemoveContainer" containerID="fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"
Jul 09 09:35:51.301573 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.301554    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8\": container with ID starting with fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8 not found: ID does not exist" containerID="fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"
Jul 09 09:35:51.301624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301577    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"} err="failed to get container status \"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8\": rpc error: code = NotFound desc = could not find container \"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8\": container with ID starting with fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8 not found: ID does not exist"
Jul 09 09:35:51.301624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301591    2576 scope.go:117] "RemoveContainer" containerID="d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"
Jul 09 09:35:51.301857 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:35:51.301842    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf\": container with ID starting with d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf not found: ID does not exist" containerID="d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"
Jul 09 09:35:51.301913 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301860    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"} err="failed to get container status \"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf\": rpc error: code = NotFound desc = could not find container \"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf\": container with ID starting with d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf not found: ID does not exist"
Jul 09 09:35:51.301913 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.301874    2576 scope.go:117] "RemoveContainer" containerID="26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"
Jul 09 09:35:51.302106 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302086    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6"} err="failed to get container status \"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6\": rpc error: code = NotFound desc = could not find container \"26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6\": container with ID starting with 26e346e023912408dc4f7bfe68a2586b50580dd58db1b4d530d34a8c186cc7b6 not found: ID does not exist"
Jul 09 09:35:51.302149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302107    2576 scope.go:117] "RemoveContainer" containerID="a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"
Jul 09 09:35:51.302356 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302333    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d"} err="failed to get container status \"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d\": rpc error: code = NotFound desc = could not find container \"a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d\": container with ID starting with a7d492d49f832cb8f82e58af9befe4e1c2c063d1f5080c9265357be2bc61a55d not found: ID does not exist"
Jul 09 09:35:51.302356 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302356    2576 scope.go:117] "RemoveContainer" containerID="927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"
Jul 09 09:35:51.302590 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302572    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e"} err="failed to get container status \"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e\": rpc error: code = NotFound desc = could not find container \"927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e\": container with ID starting with 927412be7417249aa45c42e731630929fcc1711fa2d0221a8cb8b397bfa8107e not found: ID does not exist"
Jul 09 09:35:51.302627 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302591    2576 scope.go:117] "RemoveContainer" containerID="e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"
Jul 09 09:35:51.302801 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302785    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf"} err="failed to get container status \"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf\": rpc error: code = NotFound desc = could not find container \"e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf\": container with ID starting with e4a2472d51619dd6e2d64ed17e601b62a2763ff2c61777876e9407911c3e6acf not found: ID does not exist"
Jul 09 09:35:51.302801 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302800    2576 scope.go:117] "RemoveContainer" containerID="81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"
Jul 09 09:35:51.302977 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf"} err="failed to get container status \"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf\": rpc error: code = NotFound desc = could not find container \"81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf\": container with ID starting with 81b8a0bc56f64ecdfa043ba6dc3189851347076c37f16cbb1360bf339d40becf not found: ID does not exist"
Jul 09 09:35:51.303018 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.302978    2576 scope.go:117] "RemoveContainer" containerID="fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"
Jul 09 09:35:51.303207 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.303169    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8"} err="failed to get container status \"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8\": rpc error: code = NotFound desc = could not find container \"fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8\": container with ID starting with fdbe69df02452b208fac696dcfbfba25f4e69696a4a6e1ff0c58b72a2485bab8 not found: ID does not exist"
Jul 09 09:35:51.303250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.303209    2576 scope.go:117] "RemoveContainer" containerID="d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"
Jul 09 09:35:51.303464 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.303444    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf"} err="failed to get container status \"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf\": rpc error: code = NotFound desc = could not find container \"d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf\": container with ID starting with d907e6e98d2020d55dec839aae6000e789514958f751a576ea5cf96a12cb7dbf not found: ID does not exist"
Jul 09 09:35:51.311156 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311130    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:51.311388 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311376    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311390    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="prom-label-proxy"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311404    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="prom-label-proxy"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311413    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="alertmanager"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311418    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="alertmanager"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311430    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="config-reloader"
Jul 09 09:35:51.311437 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311435    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="config-reloader"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311444    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-web"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311450    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-web"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311456    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="init-config-reloader"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311462    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="init-config-reloader"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311469    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-metric"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311475    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-metric"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311515    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="config-reloader"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311522    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-web"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311527    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311532    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="kube-rbac-proxy-metric"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311538    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="prom-label-proxy"
Jul 09 09:35:51.313129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.311543    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" containerName="alertmanager"
Jul 09 09:35:51.315973 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.315957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.318748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.318730    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 09:35:51.318839 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.318769    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 09:35:51.318893 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.318773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 09:35:51.319100 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319081    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 09:35:51.319151 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319088    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 09:35:51.319218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319148    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 09:35:51.319218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319173    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 09:35:51.319218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319098    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t74jt\""
Jul 09 09:35:51.319354 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.319211    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 09:35:51.324522 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.324502    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 09:35:51.330856 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.330831    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:51.444204 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444266 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444309 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444309 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-out\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444372 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444372 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-web-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444443 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444443 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69g8\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-kube-api-access-s69g8\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444443 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444531 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444531 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444531 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.444612 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.444528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.540333 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.540281    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca280d6-0370-4693-be2c-f495f57bf07e" path="/var/lib/kubelet/pods/3ca280d6-0370-4693-be2c-f495f57bf07e/volumes"
Jul 09 09:35:51.545641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-out\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-web-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69g8\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-kube-api-access-s69g8\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.545992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.545929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.548084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.547134    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.548084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.547748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea86ffd-adcb-4790-9571-7354d5e752a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.548658 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.548496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-out\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.548776 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.548751    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.548854 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.548831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.549064 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.549008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.549194 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.549113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ea86ffd-adcb-4790-9571-7354d5e752a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.549272 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.549238    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.549459 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.549442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.549616 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.549596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-web-config\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.550393 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.550371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.550593 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.550574    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ea86ffd-adcb-4790-9571-7354d5e752a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.557673 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.557651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69g8\" (UniqueName: \"kubernetes.io/projected/8ea86ffd-adcb-4790-9571-7354d5e752a3-kube-api-access-s69g8\") pod \"alertmanager-main-0\" (UID: \"8ea86ffd-adcb-4790-9571-7354d5e752a3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.626218 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.626198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 09:35:51.757940 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:51.757917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 09:35:51.759570 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:35:51.759545    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea86ffd_adcb_4790_9571_7354d5e752a3.slice/crio-95b8dcd4e6674d9b57007e076d865a7dd2c147307b2ef336a4ee14d29ee3a19f WatchSource:0}: Error finding container 95b8dcd4e6674d9b57007e076d865a7dd2c147307b2ef336a4ee14d29ee3a19f: Status 404 returned error can't find the container with id 95b8dcd4e6674d9b57007e076d865a7dd2c147307b2ef336a4ee14d29ee3a19f
Jul 09 09:35:52.262642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:52.262607    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ea86ffd-adcb-4790-9571-7354d5e752a3" containerID="8ccfb782af46c73e37d54f28728c67520a769d4b15a2339c5fd3b29003789e47" exitCode=0
Jul 09 09:35:52.262810 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:52.262692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerDied","Data":"8ccfb782af46c73e37d54f28728c67520a769d4b15a2339c5fd3b29003789e47"}
Jul 09 09:35:52.262810 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:52.262782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"95b8dcd4e6674d9b57007e076d865a7dd2c147307b2ef336a4ee14d29ee3a19f"}
Jul 09 09:35:53.269909 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.269877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"7bbc2a4c6bd203e333407251214d92500aa3025b20b3e56045382f305b235ec5"}
Jul 09 09:35:53.270276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.269940    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"df75c9d304b49cbbbde659f6a2cd0b51ad43166c9e319c948cff0f5f6940a0b9"}
Jul 09 09:35:53.270276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.269966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"409e740aecba994d6f5e91588f44729eed9ab585da8cb288757924480cc97c82"}
Jul 09 09:35:53.270276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.269991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"fbfe479e75233ef3ae40623add35ef2cc6e2c0f5ef8cdde3702f22aae4758988"}
Jul 09 09:35:53.270276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.270019    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"c1a7ee75cfa200eba757efb2df85ff2e15f0f177350c09be57d6788ae1b85d01"}
Jul 09 09:35:53.270276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.270042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ea86ffd-adcb-4790-9571-7354d5e752a3","Type":"ContainerStarted","Data":"56c8655be68cd0e5612dba18ef6a670a7ff29f10a16727e84aa5355dd2107780"}
Jul 09 09:35:53.298476 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.298420    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.28445164 podStartE2EDuration="2.298400431s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.948791ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:35:53.296655819 +0000 UTC m=+138.332465545" watchObservedRunningTime="2026-07-09 09:35:53.298400431 +0000 UTC m=+138.334210139"
Jul 09 09:35:53.397358 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.397323    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"]
Jul 09 09:35:53.399506 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.399491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.402414 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402396    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 09:35:53.402414 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402409    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 09:35:53.402581 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402496    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4j6l4\""
Jul 09 09:35:53.402581 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 09:35:53.402758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402745    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 09:35:53.402801 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.402742    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 09:35:53.410004 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.409984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 09:35:53.421634 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.421612    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"]
Jul 09 09:35:53.463917 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.463893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.463922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-federate-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.463956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.464013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464136 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.464069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk48f\" (UniqueName: \"kubernetes.io/projected/91dea31f-f28c-41f3-9807-a5c0e47a8d49-kube-api-access-jk48f\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464136 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.464091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-metrics-client-ca\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464136 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.464108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.464245 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.464171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-serving-certs-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-serving-certs-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-federate-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk48f\" (UniqueName: \"kubernetes.io/projected/91dea31f-f28c-41f3-9807-a5c0e47a8d49-kube-api-access-jk48f\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565607 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-metrics-client-ca\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.565764 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.565740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.566066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.566043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-serving-certs-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.566170 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.566143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-metrics-client-ca\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.566876 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.566839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.568207 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.568169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.568385 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.568366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-federate-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.568426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.568391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-telemeter-client-tls\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.568426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.568391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91dea31f-f28c-41f3-9807-a5c0e47a8d49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.576593 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.576566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk48f\" (UniqueName: \"kubernetes.io/projected/91dea31f-f28c-41f3-9807-a5c0e47a8d49-kube-api-access-jk48f\") pod \"telemeter-client-5d7c4c975c-6rvkn\" (UID: \"91dea31f-f28c-41f3-9807-a5c0e47a8d49\") " pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.708952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.708928    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"
Jul 09 09:35:53.837976 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:53.837955    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn"]
Jul 09 09:35:53.839841 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:35:53.839816    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dea31f_f28c_41f3_9807_a5c0e47a8d49.slice/crio-dd1eaee792d2f3c5d8c9d6614c640f4e7a97865b6c87458498a3a2e50f775798 WatchSource:0}: Error finding container dd1eaee792d2f3c5d8c9d6614c640f4e7a97865b6c87458498a3a2e50f775798: Status 404 returned error can't find the container with id dd1eaee792d2f3c5d8c9d6614c640f4e7a97865b6c87458498a3a2e50f775798
Jul 09 09:35:54.274911 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:54.274874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn" event={"ID":"91dea31f-f28c-41f3-9807-a5c0e47a8d49","Type":"ContainerStarted","Data":"dd1eaee792d2f3c5d8c9d6614c640f4e7a97865b6c87458498a3a2e50f775798"}
Jul 09 09:35:56.284456 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:56.284422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn" event={"ID":"91dea31f-f28c-41f3-9807-a5c0e47a8d49","Type":"ContainerStarted","Data":"c58cce50ca51e9fc7a73df95846412860a71de9ab6435fc82156687786e510e9"}
Jul 09 09:35:56.284787 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:56.284477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn" event={"ID":"91dea31f-f28c-41f3-9807-a5c0e47a8d49","Type":"ContainerStarted","Data":"6a77dacc135d976ec32b61be7ba164ed90603c52225dacc86452628acad05cbf"}
Jul 09 09:35:56.284787 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:56.284496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn" event={"ID":"91dea31f-f28c-41f3-9807-a5c0e47a8d49","Type":"ContainerStarted","Data":"620df94c5489089a6263cc6653ca64784a4087b13232c8cf6164a25799494218"}
Jul 09 09:35:56.309107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:56.309062    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d7c4c975c-6rvkn" podStartSLOduration=1.567450065 podStartE2EDuration="3.309047134s" totalImagesPullingTime="1.741597069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:35:56.307599448 +0000 UTC m=+141.343409169" watchObservedRunningTime="2026-07-09 09:35:56.309047134 +0000 UTC m=+141.344856838"
Jul 09 09:35:59.508809 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.508762    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:35:59.511370 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.511347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.522296 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.522271    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:35:59.562169 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.562138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:59.616613 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616713 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616713 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9z45\" (UniqueName: \"kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616713 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.616813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.616772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.628144 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.628118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:59.707975 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.707951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:59.717044 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717139 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717139 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9z45\" (UniqueName: \"kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717139 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717829 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717800    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.717829 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717806    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.718016 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.717996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.718063 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.718046    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.719937 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.719905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.719937 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.719905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.726510 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.726485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9z45\" (UniqueName: \"kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45\") pod \"console-856866fd98-84qtc\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") " pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.765832 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.765771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:35:59.821586 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.821553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:35:59.974074 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:35:59.974047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:35:59.977233 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:35:59.977202    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0b469f_d14b_4027_b036_abc45c702807.slice/crio-17fcde698add884a8f8add6f20806991fc077bc0b43b3f21080035ff50a8cfdc WatchSource:0}: Error finding container 17fcde698add884a8f8add6f20806991fc077bc0b43b3f21080035ff50a8cfdc: Status 404 returned error can't find the container with id 17fcde698add884a8f8add6f20806991fc077bc0b43b3f21080035ff50a8cfdc
Jul 09 09:36:00.301336 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:00.301233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856866fd98-84qtc" event={"ID":"3c0b469f-d14b-4027-b036-abc45c702807","Type":"ContainerStarted","Data":"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"}
Jul 09 09:36:00.301336 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:00.301296    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856866fd98-84qtc" event={"ID":"3c0b469f-d14b-4027-b036-abc45c702807","Type":"ContainerStarted","Data":"17fcde698add884a8f8add6f20806991fc077bc0b43b3f21080035ff50a8cfdc"}
Jul 09 09:36:00.303649 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:00.303624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:00.319234 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:00.319161    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-856866fd98-84qtc" podStartSLOduration=1.319146744 podStartE2EDuration="1.319146744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:36:00.317849609 +0000 UTC m=+145.353659313" watchObservedRunningTime="2026-07-09 09:36:00.319146744 +0000 UTC m=+145.354956448"
Jul 09 09:36:00.405103 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:00.405069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:09.822621 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:09.822581    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:36:09.823101 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:09.822643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:36:09.827659 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:09.827640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:36:10.343243 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:10.343215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:36:10.390448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:10.390413    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:36:10.528137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:10.528110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:10.625058 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:10.624987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:10.725710 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:10.725677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:35.411342 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.411295    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6b6b9684f6-mx7q9" podUID="b7b98831-85e9-4d85-9736-cfa863a9cabc" containerName="console" containerID="cri-o://ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b" gracePeriod=15
Jul 09 09:36:35.650611 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.650588    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:36:35.705787 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.705943 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705810    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.705943 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705842    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.705943 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705877    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs4cq\" (UniqueName: \"kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.705943 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705928    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.706151 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.705974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.706151 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.706009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca\") pod \"b7b98831-85e9-4d85-9736-cfa863a9cabc\" (UID: \"b7b98831-85e9-4d85-9736-cfa863a9cabc\") "
Jul 09 09:36:35.706304 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.706276    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:36:35.706367 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.706285    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:36:35.706470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.706443    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config" (OuterVolumeSpecName: "console-config") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:36:35.706551 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.706529    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:36:35.708082 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.708062    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:36:35.708171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.708118    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:36:35.708171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.708157    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq" (OuterVolumeSpecName: "kube-api-access-vs4cq") pod "b7b98831-85e9-4d85-9736-cfa863a9cabc" (UID: "b7b98831-85e9-4d85-9736-cfa863a9cabc"). InnerVolumeSpecName "kube-api-access-vs4cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:36:35.806642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806613    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806641    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-service-ca\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806653    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806667    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-oauth-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806679    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7b98831-85e9-4d85-9736-cfa863a9cabc-trusted-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806690    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4cq\" (UniqueName: \"kubernetes.io/projected/b7b98831-85e9-4d85-9736-cfa863a9cabc-kube-api-access-vs4cq\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:35.806882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:35.806701    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7b98831-85e9-4d85-9736-cfa863a9cabc-console-oauth-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:36:36.440799 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.440756    2576 generic.go:356] "Generic (PLEG): container finished" podID="b7b98831-85e9-4d85-9736-cfa863a9cabc" containerID="ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b" exitCode=0
Jul 09 09:36:36.441225 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.440803    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6b9684f6-mx7q9" event={"ID":"b7b98831-85e9-4d85-9736-cfa863a9cabc","Type":"ContainerDied","Data":"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"}
Jul 09 09:36:36.441225 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.440820    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6b9684f6-mx7q9"
Jul 09 09:36:36.441225 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.440887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6b9684f6-mx7q9" event={"ID":"b7b98831-85e9-4d85-9736-cfa863a9cabc","Type":"ContainerDied","Data":"a0143036111ec933e6f7ddc73c698a27f7dd06b379181df1f89c8068f7b6a2a2"}
Jul 09 09:36:36.441225 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.440916    2576 scope.go:117] "RemoveContainer" containerID="ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"
Jul 09 09:36:36.449321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.449297    2576 scope.go:117] "RemoveContainer" containerID="ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"
Jul 09 09:36:36.449618 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:36:36.449594    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b\": container with ID starting with ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b not found: ID does not exist" containerID="ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"
Jul 09 09:36:36.449686 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.449627    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b"} err="failed to get container status \"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b\": rpc error: code = NotFound desc = could not find container \"ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b\": container with ID starting with ff121c43195f75788e70409ed65b0eee36daf7ed08048044da3c6b622cd7827b not found: ID does not exist"
Jul 09 09:36:36.463099 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.463080    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:36:36.466632 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:36.466612    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6b9684f6-mx7q9"]
Jul 09 09:36:37.540985 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:37.540948    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b98831-85e9-4d85-9736-cfa863a9cabc" path="/var/lib/kubelet/pods/b7b98831-85e9-4d85-9736-cfa863a9cabc/volumes"
Jul 09 09:36:48.517766 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:48.517727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:53.376826 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:53.376786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:36:53.432372 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:36:53.432333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:03.537465 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:03.537424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:03.603855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:03.603811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:09.187712 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.187678    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:37:09.188066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.187954    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b98831-85e9-4d85-9736-cfa863a9cabc" containerName="console"
Jul 09 09:37:09.188066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.187966    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b98831-85e9-4d85-9736-cfa863a9cabc" containerName="console"
Jul 09 09:37:09.188066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.188015    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b98831-85e9-4d85-9736-cfa863a9cabc" containerName="console"
Jul 09 09:37:09.190758 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.190741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.201746 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.201712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:37:09.244009 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.243975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:09.274805 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.274903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.274903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.274903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.274999 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.275034 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.274993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.275034 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.275016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczvx\" (UniqueName: \"kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.302739 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.302710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:09.375584 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375677 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375677 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375776 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375812 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczvx\" (UniqueName: \"kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375868 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.375913 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.375885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.376521 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.376499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.376607 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.376582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.376648 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.376607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.376690 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.376668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.378642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.378617    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.378740 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.378626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.384467 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.384444    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczvx\" (UniqueName: \"kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx\") pod \"console-5685ddcd95-lk7pm\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") " pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.499106 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.499036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:09.625072 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:09.625049    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:37:09.627525 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:37:09.627501    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90247838_f4d2_4016_94df_4ca9e93472cf.slice/crio-44b752928735bd3680ce50530feb546be087cdd0d22a020c7bd04a12b86868f4 WatchSource:0}: Error finding container 44b752928735bd3680ce50530feb546be087cdd0d22a020c7bd04a12b86868f4: Status 404 returned error can't find the container with id 44b752928735bd3680ce50530feb546be087cdd0d22a020c7bd04a12b86868f4
Jul 09 09:37:10.565898 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:10.565866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5685ddcd95-lk7pm" event={"ID":"90247838-f4d2-4016-94df-4ca9e93472cf","Type":"ContainerStarted","Data":"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"}
Jul 09 09:37:10.566284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:10.565910    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5685ddcd95-lk7pm" event={"ID":"90247838-f4d2-4016-94df-4ca9e93472cf","Type":"ContainerStarted","Data":"44b752928735bd3680ce50530feb546be087cdd0d22a020c7bd04a12b86868f4"}
Jul 09 09:37:10.585498 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:10.585455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5685ddcd95-lk7pm" podStartSLOduration=1.5854402570000001 podStartE2EDuration="1.585440257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:37:10.583404309 +0000 UTC m=+215.619214025" watchObservedRunningTime="2026-07-09 09:37:10.585440257 +0000 UTC m=+215.621249962"
Jul 09 09:37:15.493956 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:15.493922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:15.551636 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:15.551605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:19.499292 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.499256    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:19.499797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.499350    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:19.504128 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.504105    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:19.601252 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.601226    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:37:19.648707 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.648679    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:37:19.761066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.760977    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:19.817708 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.817673    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:19.874861 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:19.874834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:28.733163 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:28.733100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:31.608394 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:31.608362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:31.709550 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:31.709508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:31.810260 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:31.810225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:38.017026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:38.016975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:38.098462 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:38.098429    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:44.669723 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.669681    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-856866fd98-84qtc" podUID="3c0b469f-d14b-4027-b036-abc45c702807" containerName="console" containerID="cri-o://28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87" gracePeriod=15
Jul 09 09:37:44.907571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.907548    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:37:44.922276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922223    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9z45\" (UniqueName: \"kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922409 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922409 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922409 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922409 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922608 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922509    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle\") pod \"3c0b469f-d14b-4027-b036-abc45c702807\" (UID: \"3c0b469f-d14b-4027-b036-abc45c702807\") "
Jul 09 09:37:44.922778 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922733    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config" (OuterVolumeSpecName: "console-config") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:37:44.922778 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922742    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca" (OuterVolumeSpecName: "service-ca") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:37:44.923011 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.922785    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:37:44.923061 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.923047    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:37:44.924624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.924600    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:37:44.925055 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.925024    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:37:44.925154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:44.925036    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45" (OuterVolumeSpecName: "kube-api-access-x9z45") pod "3c0b469f-d14b-4027-b036-abc45c702807" (UID: "3c0b469f-d14b-4027-b036-abc45c702807"). InnerVolumeSpecName "kube-api-access-x9z45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:37:45.023084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023059    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023081    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-console-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023097    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-oauth-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023109    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c0b469f-d14b-4027-b036-abc45c702807-console-oauth-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023119    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-service-ca\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023127    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c0b469f-d14b-4027-b036-abc45c702807-trusted-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.023253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.023135    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9z45\" (UniqueName: \"kubernetes.io/projected/3c0b469f-d14b-4027-b036-abc45c702807-kube-api-access-x9z45\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:37:45.696594 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.696562    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c0b469f-d14b-4027-b036-abc45c702807" containerID="28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87" exitCode=0
Jul 09 09:37:45.696962 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.696606    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856866fd98-84qtc"
Jul 09 09:37:45.696962 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.696650    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856866fd98-84qtc" event={"ID":"3c0b469f-d14b-4027-b036-abc45c702807","Type":"ContainerDied","Data":"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"}
Jul 09 09:37:45.696962 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.696708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856866fd98-84qtc" event={"ID":"3c0b469f-d14b-4027-b036-abc45c702807","Type":"ContainerDied","Data":"17fcde698add884a8f8add6f20806991fc077bc0b43b3f21080035ff50a8cfdc"}
Jul 09 09:37:45.696962 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.696734    2576 scope.go:117] "RemoveContainer" containerID="28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"
Jul 09 09:37:45.704796 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.704779    2576 scope.go:117] "RemoveContainer" containerID="28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"
Jul 09 09:37:45.705062 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:37:45.705035    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87\": container with ID starting with 28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87 not found: ID does not exist" containerID="28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"
Jul 09 09:37:45.705163 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.705072    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87"} err="failed to get container status \"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87\": rpc error: code = NotFound desc = could not find container \"28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87\": container with ID starting with 28bd37399077782630b8dbc91aea8211b0f42d210a287c50c189bfb2d98dbc87 not found: ID does not exist"
Jul 09 09:37:45.736298 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.736274    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:37:45.747016 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:45.746994    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-856866fd98-84qtc"]
Jul 09 09:37:47.540734 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:47.540705    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0b469f-d14b-4027-b036-abc45c702807" path="/var/lib/kubelet/pods/3c0b469f-d14b-4027-b036-abc45c702807/volumes"
Jul 09 09:37:55.118785 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.118753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:55.128879 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.128852    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"]
Jul 09 09:37:55.129160 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.129147    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0b469f-d14b-4027-b036-abc45c702807" containerName="console"
Jul 09 09:37:55.129160 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.129161    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0b469f-d14b-4027-b036-abc45c702807" containerName="console"
Jul 09 09:37:55.129270 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.129236    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0b469f-d14b-4027-b036-abc45c702807" containerName="console"
Jul 09 09:37:55.133425 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.133410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.137133 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.137110    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 09:37:55.137256 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.137149    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 09:37:55.137256 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.137195    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 09:37:55.137256 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.137229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 09:37:55.137426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.137273    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-klk6x\""
Jul 09 09:37:55.143919 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.143894    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"]
Jul 09 09:37:55.187584 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.187559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6cp\" (UniqueName: \"kubernetes.io/projected/4bd45cfb-8328-48af-a90b-cee5056cebc8-kube-api-access-qf6cp\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.187683 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.187601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bd45cfb-8328-48af-a90b-cee5056cebc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.220040 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.220010    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"]
Jul 09 09:37:55.223104 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.223088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.223917 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.223896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:55.226191 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.226161    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 09:37:55.238768 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.238748    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"]
Jul 09 09:37:55.288630 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.288608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bd45cfb-8328-48af-a90b-cee5056cebc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.288741 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.288647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltd87\" (UniqueName: \"kubernetes.io/projected/fe1ff2ca-2bc2-4564-a559-58df926bd153-kube-api-access-ltd87\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.288741 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.288684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe1ff2ca-2bc2-4564-a559-58df926bd153-klusterlet-config\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.288741 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.288707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6cp\" (UniqueName: \"kubernetes.io/projected/4bd45cfb-8328-48af-a90b-cee5056cebc8-kube-api-access-qf6cp\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.288859 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.288751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe1ff2ca-2bc2-4564-a559-58df926bd153-tmp\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.290625 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.290601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:37:55.291278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.291263    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bd45cfb-8328-48af-a90b-cee5056cebc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.298133 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.298111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6cp\" (UniqueName: \"kubernetes.io/projected/4bd45cfb-8328-48af-a90b-cee5056cebc8-kube-api-access-qf6cp\") pod \"managed-serviceaccount-addon-agent-74c8b55688-wm7lt\" (UID: \"4bd45cfb-8328-48af-a90b-cee5056cebc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.389994 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.389942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltd87\" (UniqueName: \"kubernetes.io/projected/fe1ff2ca-2bc2-4564-a559-58df926bd153-kube-api-access-ltd87\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.389994 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.389991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe1ff2ca-2bc2-4564-a559-58df926bd153-klusterlet-config\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.390103 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.390035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe1ff2ca-2bc2-4564-a559-58df926bd153-tmp\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.390416 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.390391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe1ff2ca-2bc2-4564-a559-58df926bd153-tmp\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.392228 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.392209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe1ff2ca-2bc2-4564-a559-58df926bd153-klusterlet-config\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.399866 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.399846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltd87\" (UniqueName: \"kubernetes.io/projected/fe1ff2ca-2bc2-4564-a559-58df926bd153-kube-api-access-ltd87\") pod \"klusterlet-addon-workmgr-795fffc99-w5lqs\" (UID: \"fe1ff2ca-2bc2-4564-a559-58df926bd153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.451159 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.451136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"
Jul 09 09:37:55.531507 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.531482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:55.580064 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.579987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt"]
Jul 09 09:37:55.583573 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:37:55.583550    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bd45cfb_8328_48af_a90b_cee5056cebc8.slice/crio-4e83c2bd3f090f9394563ced98765abfd248a5941694d6825f0c7afa31f7b55b WatchSource:0}: Error finding container 4e83c2bd3f090f9394563ced98765abfd248a5941694d6825f0c7afa31f7b55b: Status 404 returned error can't find the container with id 4e83c2bd3f090f9394563ced98765abfd248a5941694d6825f0c7afa31f7b55b
Jul 09 09:37:55.664428 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.664368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"]
Jul 09 09:37:55.667025 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:37:55.667001    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1ff2ca_2bc2_4564_a559_58df926bd153.slice/crio-5f46a4b23529faccc1a7c3310c4455049d10555c174da1ff74f4adc4b698473d WatchSource:0}: Error finding container 5f46a4b23529faccc1a7c3310c4455049d10555c174da1ff74f4adc4b698473d: Status 404 returned error can't find the container with id 5f46a4b23529faccc1a7c3310c4455049d10555c174da1ff74f4adc4b698473d
Jul 09 09:37:55.733161 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.733132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs" event={"ID":"fe1ff2ca-2bc2-4564-a559-58df926bd153","Type":"ContainerStarted","Data":"5f46a4b23529faccc1a7c3310c4455049d10555c174da1ff74f4adc4b698473d"}
Jul 09 09:37:55.734201 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:55.734160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt" event={"ID":"4bd45cfb-8328-48af-a90b-cee5056cebc8","Type":"ContainerStarted","Data":"4e83c2bd3f090f9394563ced98765abfd248a5941694d6825f0c7afa31f7b55b"}
Jul 09 09:37:59.755408 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.755372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs" event={"ID":"fe1ff2ca-2bc2-4564-a559-58df926bd153","Type":"ContainerStarted","Data":"e14777ca5e385d9643d5d5682e1924e3d9b04a641425323e7bc3237e004b0886"}
Jul 09 09:37:59.755855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.755552    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:59.757149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.757122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt" event={"ID":"4bd45cfb-8328-48af-a90b-cee5056cebc8","Type":"ContainerStarted","Data":"6183c2f61e5cdc22938d5ad59507b0a54b9fa317deff6f9b74c7356110dfe4de"}
Jul 09 09:37:59.757316 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.757303    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs"
Jul 09 09:37:59.773149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.773097    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795fffc99-w5lqs" podStartSLOduration=0.894361685 podStartE2EDuration="4.773080865s" totalImagesPullingTime="3.87871918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:37:59.771826247 +0000 UTC m=+264.807635952" watchObservedRunningTime="2026-07-09 09:37:59.773080865 +0000 UTC m=+264.808890570"
Jul 09 09:37:59.787005 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:37:59.786891    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74c8b55688-wm7lt" podStartSLOduration=0.835952607 podStartE2EDuration="4.786880375s" totalImagesPullingTime="3.950927768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:37:59.786857632 +0000 UTC m=+264.822667339" watchObservedRunningTime="2026-07-09 09:37:59.786880375 +0000 UTC m=+264.822690104"
Jul 09 09:38:10.018942 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:10.018900    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:38:10.078823 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:10.078797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:38:35.347732 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:35.347708    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 09:38:43.944253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:43.944213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:38:45.405320 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.405290    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"]
Jul 09 09:38:45.407529 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.407514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.410200 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.410162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 09:38:45.410448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.410433    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b6wmd\""
Jul 09 09:38:45.411388 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.411373    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 09:38:45.420016 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.419999    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"]
Jul 09 09:38:45.575235 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.575200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrmr\" (UniqueName: \"kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.575405 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.575253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.575405 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.575289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.676668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.676575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrmr\" (UniqueName: \"kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.676668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.676634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.676668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.676659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.677026 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.677006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.677088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.677075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.685490 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.685467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrmr\" (UniqueName: \"kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.716383 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.716361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:38:45.840378 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.840355    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"]
Jul 09 09:38:45.842743 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:38:45.842713    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab4993c_2eed_48ad_b271_161c5df96d0a.slice/crio-29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4 WatchSource:0}: Error finding container 29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4: Status 404 returned error can't find the container with id 29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4
Jul 09 09:38:45.844729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.844711    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:38:45.925159 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:45.925131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerStarted","Data":"29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4"}
Jul 09 09:38:47.959340 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:47.959302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:38:51.950471 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:51.950440    2576 generic.go:356] "Generic (PLEG): container finished" podID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerID="f475a0a9516708ae9f52ad923a49800ec18e55eb6e303f7bf52d7de07a9e3903" exitCode=0
Jul 09 09:38:51.950855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:51.950478    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerDied","Data":"f475a0a9516708ae9f52ad923a49800ec18e55eb6e303f7bf52d7de07a9e3903"}
Jul 09 09:38:53.960762 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:53.960726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerStarted","Data":"ab2aaa71b5570463a91d863a919d8aa2a3edb9de61b82375adcd49f419a0e756"}
Jul 09 09:38:54.966729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:54.966693    2576 generic.go:356] "Generic (PLEG): container finished" podID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerID="ab2aaa71b5570463a91d863a919d8aa2a3edb9de61b82375adcd49f419a0e756" exitCode=0
Jul 09 09:38:54.967102 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:38:54.966735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerDied","Data":"ab2aaa71b5570463a91d863a919d8aa2a3edb9de61b82375adcd49f419a0e756"}
Jul 09 09:39:02.000603 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:02.000568    2576 generic.go:356] "Generic (PLEG): container finished" podID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerID="e6e7a22d4abd80b8935be75e289ac2e3335d1d57970464bf5602b7863e7f5b85" exitCode=0
Jul 09 09:39:02.000973 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:02.000637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerDied","Data":"e6e7a22d4abd80b8935be75e289ac2e3335d1d57970464bf5602b7863e7f5b85"}
Jul 09 09:39:03.127878 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.127854    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:39:03.227327 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.227302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnrmr\" (UniqueName: \"kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr\") pod \"bab4993c-2eed-48ad-b271-161c5df96d0a\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") "
Jul 09 09:39:03.227454 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.227356    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util\") pod \"bab4993c-2eed-48ad-b271-161c5df96d0a\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") "
Jul 09 09:39:03.227454 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.227411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle\") pod \"bab4993c-2eed-48ad-b271-161c5df96d0a\" (UID: \"bab4993c-2eed-48ad-b271-161c5df96d0a\") "
Jul 09 09:39:03.228028 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.228003    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle" (OuterVolumeSpecName: "bundle") pod "bab4993c-2eed-48ad-b271-161c5df96d0a" (UID: "bab4993c-2eed-48ad-b271-161c5df96d0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:39:03.229420 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.229391    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr" (OuterVolumeSpecName: "kube-api-access-rnrmr") pod "bab4993c-2eed-48ad-b271-161c5df96d0a" (UID: "bab4993c-2eed-48ad-b271-161c5df96d0a"). InnerVolumeSpecName "kube-api-access-rnrmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:39:03.231860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.231839    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util" (OuterVolumeSpecName: "util") pod "bab4993c-2eed-48ad-b271-161c5df96d0a" (UID: "bab4993c-2eed-48ad-b271-161c5df96d0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:39:03.328250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.328159    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnrmr\" (UniqueName: \"kubernetes.io/projected/bab4993c-2eed-48ad-b271-161c5df96d0a-kube-api-access-rnrmr\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:39:03.328250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.328215    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-util\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:39:03.328250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:03.328225    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab4993c-2eed-48ad-b271-161c5df96d0a-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:39:04.010585 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:04.010547    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz" event={"ID":"bab4993c-2eed-48ad-b271-161c5df96d0a","Type":"ContainerDied","Data":"29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4"}
Jul 09 09:39:04.010757 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:04.010598    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a621ea24f87a00a1fff3f470954bec25a29005b67edc4752c68d0aedad38b4"
Jul 09 09:39:04.010757 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:04.010557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589khnzz"
Jul 09 09:39:06.435551 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:06.435519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:06.535775 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:06.535741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:07.653990 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.653956    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"]
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654282    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="pull"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654294    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="pull"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654306    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="util"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654313    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="util"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654322    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="extract"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654327    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="extract"
Jul 09 09:39:07.654466 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.654378    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab4993c-2eed-48ad-b271-161c5df96d0a" containerName="extract"
Jul 09 09:39:07.686824 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.686794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.690014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.689996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 09:39:07.690575 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.690556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 09:39:07.690912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.690896    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 09:39:07.696567 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.696546    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-h8vsz\""
Jul 09 09:39:07.698722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.698700    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"]
Jul 09 09:39:07.767838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.767799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfdh\" (UniqueName: \"kubernetes.io/projected/24ce7327-8eaf-444b-ac4f-91afb7c807cf-kube-api-access-zmfdh\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.767838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.767839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24ce7327-8eaf-444b-ac4f-91afb7c807cf-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.868406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.868379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfdh\" (UniqueName: \"kubernetes.io/projected/24ce7327-8eaf-444b-ac4f-91afb7c807cf-kube-api-access-zmfdh\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.868502 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.868414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24ce7327-8eaf-444b-ac4f-91afb7c807cf-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.870780 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.870757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24ce7327-8eaf-444b-ac4f-91afb7c807cf-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.877693 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.877662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfdh\" (UniqueName: \"kubernetes.io/projected/24ce7327-8eaf-444b-ac4f-91afb7c807cf-kube-api-access-zmfdh\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf\" (UID: \"24ce7327-8eaf-444b-ac4f-91afb7c807cf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:07.997314 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:07.997282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:08.152980 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:08.152944    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"]
Jul 09 09:39:08.156594 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:39:08.156559    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ce7327_8eaf_444b_ac4f_91afb7c807cf.slice/crio-0a5dd5d33d08141a74925e6710eb29892e81e830666f8a80369393b035351246 WatchSource:0}: Error finding container 0a5dd5d33d08141a74925e6710eb29892e81e830666f8a80369393b035351246: Status 404 returned error can't find the container with id 0a5dd5d33d08141a74925e6710eb29892e81e830666f8a80369393b035351246
Jul 09 09:39:09.030120 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:09.030082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf" event={"ID":"24ce7327-8eaf-444b-ac4f-91afb7c807cf","Type":"ContainerStarted","Data":"0a5dd5d33d08141a74925e6710eb29892e81e830666f8a80369393b035351246"}
Jul 09 09:39:12.045939 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.045900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf" event={"ID":"24ce7327-8eaf-444b-ac4f-91afb7c807cf","Type":"ContainerStarted","Data":"e8fb951749c54b50e4e5c526cb1ae54f5c15ef2b245038e846c77c5aed5063b0"}
Jul 09 09:39:12.046508 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.046026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:12.052022 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.051984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:12.068755 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.068717    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf" podStartSLOduration=1.763380795 podStartE2EDuration="5.068703593s" totalImagesPullingTime="3.305322798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:39:12.067894705 +0000 UTC m=+337.103704410" watchObservedRunningTime="2026-07-09 09:39:12.068703593 +0000 UTC m=+337.104513298"
Jul 09 09:39:12.563440 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.563403    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:12.666127 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.666087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:12.800435 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:12.800390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:15.292827 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:15.292789    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:15.386133 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:15.386085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:16.358760 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:16.358719    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:16.463662 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:16.463634    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:32.135877 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:32.135776    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:32.194633 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:32.194591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:33.053312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:33.053279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g6hxf"
Jul 09 09:39:33.899735 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:33.899687    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:39:34.000695 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:39:34.000657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:20.975384 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:20.975346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.041642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.041608    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:21.045059 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.045040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.048608 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.048588    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 09:40:21.049479 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.049457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 09 09:40:21.049599 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.049497    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tstqm\""
Jul 09 09:40:21.049664 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.049609    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 09:40:21.049814 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.049793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.052123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.051787    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"]
Jul 09 09:40:21.054904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.054884    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.061448 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.056599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:21.062132 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.062110    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 09:40:21.062642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.062156    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6gqkr\""
Jul 09 09:40:21.073395 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.073377    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"]
Jul 09 09:40:21.139788 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.139766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rl4s\" (UniqueName: \"kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.139882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.139808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlzx\" (UniqueName: \"kubernetes.io/projected/c9d2a769-a344-4c12-bd9b-0259fa8d0748-kube-api-access-5jlzx\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.139882 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.139832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9d2a769-a344-4c12-bd9b-0259fa8d0748-cert\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.139952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.139902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.149024 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.148995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.241016 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.240963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlzx\" (UniqueName: \"kubernetes.io/projected/c9d2a769-a344-4c12-bd9b-0259fa8d0748-kube-api-access-5jlzx\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.241016 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.240996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9d2a769-a344-4c12-bd9b-0259fa8d0748-cert\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.241137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.241022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.241137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.241047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rl4s\" (UniqueName: \"kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.243567 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.243546    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.243630 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.243616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9d2a769-a344-4c12-bd9b-0259fa8d0748-cert\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.246632 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.246605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.248956 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.248926    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rl4s\" (UniqueName: \"kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s\") pod \"kserve-controller-manager-5876b4f96d-vzq2b\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") " pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.249040 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.248988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlzx\" (UniqueName: \"kubernetes.io/projected/c9d2a769-a344-4c12-bd9b-0259fa8d0748-kube-api-access-5jlzx\") pod \"llmisvc-controller-manager-67d95fdc48-rqnv4\" (UID: \"c9d2a769-a344-4c12-bd9b-0259fa8d0748\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.346661 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.346635    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.355543 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.355526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:21.369364 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.369343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:21.491457 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.491361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:21.494052 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:40:21.494016    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6ac0b5_0697_40d7_925c_68ff26e2f2bc.slice/crio-e5cb9ab540d114467d7592498727a5ab081c982f830fea84efbef5d760ae8bc2 WatchSource:0}: Error finding container e5cb9ab540d114467d7592498727a5ab081c982f830fea84efbef5d760ae8bc2: Status 404 returned error can't find the container with id e5cb9ab540d114467d7592498727a5ab081c982f830fea84efbef5d760ae8bc2
Jul 09 09:40:21.512880 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.512851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"]
Jul 09 09:40:21.515234 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:40:21.515208    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9d2a769_a344_4c12_bd9b_0259fa8d0748.slice/crio-2ec7e60b4912c5f73759275d90916a477b1adc11575c9f052100a7188f0a4bee WatchSource:0}: Error finding container 2ec7e60b4912c5f73759275d90916a477b1adc11575c9f052100a7188f0a4bee: Status 404 returned error can't find the container with id 2ec7e60b4912c5f73759275d90916a477b1adc11575c9f052100a7188f0a4bee
Jul 09 09:40:21.526623 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.526586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.627320 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.627296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:21.728140 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:21.728116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:40:22.356658 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:22.356618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" event={"ID":"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc","Type":"ContainerStarted","Data":"e5cb9ab540d114467d7592498727a5ab081c982f830fea84efbef5d760ae8bc2"}
Jul 09 09:40:22.359120 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:22.359086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4" event={"ID":"c9d2a769-a344-4c12-bd9b-0259fa8d0748","Type":"ContainerStarted","Data":"2ec7e60b4912c5f73759275d90916a477b1adc11575c9f052100a7188f0a4bee"}
Jul 09 09:40:25.376645 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.376612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" event={"ID":"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc","Type":"ContainerStarted","Data":"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"}
Jul 09 09:40:25.377098 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.376712    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:25.378086 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.378063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4" event={"ID":"c9d2a769-a344-4c12-bd9b-0259fa8d0748","Type":"ContainerStarted","Data":"6d54d8fe26e07c096356a664948771a57a7456fd9a19c1f068dc15f4c3132050"}
Jul 09 09:40:25.378220 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.378108    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:25.397133 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.397087    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" podStartSLOduration=1.103817206 podStartE2EDuration="4.397073565s" totalImagesPullingTime="3.293256359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:40:25.395583696 +0000 UTC m=+410.431393401" watchObservedRunningTime="2026-07-09 09:40:25.397073565 +0000 UTC m=+410.432883270"
Jul 09 09:40:25.417582 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:25.417544    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4" podStartSLOduration=1.093604819 podStartE2EDuration="4.417532316s" totalImagesPullingTime="3.323927497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:40:25.415920016 +0000 UTC m=+410.451729721" watchObservedRunningTime="2026-07-09 09:40:25.417532316 +0000 UTC m=+410.453342093"
Jul 09 09:40:56.383722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:56.383684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-rqnv4"
Jul 09 09:40:56.386790 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:56.386766    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:57.603798 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.603724    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:57.604210 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.603869    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" podUID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" containerName="manager" containerID="cri-o://0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd" gracePeriod=10
Jul 09 09:40:57.627547 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.627524    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-htq6x"]
Jul 09 09:40:57.632302 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.632275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.641338 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.641320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-htq6x"]
Jul 09 09:40:57.728419 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.728397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55cb3d18-767c-44a4-b768-11c9acaa6545-cert\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.728507 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.728427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfksv\" (UniqueName: \"kubernetes.io/projected/55cb3d18-767c-44a4-b768-11c9acaa6545-kube-api-access-bfksv\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.828803 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.828771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfksv\" (UniqueName: \"kubernetes.io/projected/55cb3d18-767c-44a4-b768-11c9acaa6545-kube-api-access-bfksv\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.828945 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.828891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55cb3d18-767c-44a4-b768-11c9acaa6545-cert\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.831074 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.831050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55cb3d18-767c-44a4-b768-11c9acaa6545-cert\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.838528 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.838505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfksv\" (UniqueName: \"kubernetes.io/projected/55cb3d18-767c-44a4-b768-11c9acaa6545-kube-api-access-bfksv\") pod \"kserve-controller-manager-5876b4f96d-htq6x\" (UID: \"55cb3d18-767c-44a4-b768-11c9acaa6545\") " pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:57.873807 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.873762    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:57.996358 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:57.996326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:58.030338 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.030315    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rl4s\" (UniqueName: \"kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s\") pod \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") "
Jul 09 09:40:58.030468 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.030351    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert\") pod \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\" (UID: \"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc\") "
Jul 09 09:40:58.032686 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.032658    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert" (OuterVolumeSpecName: "cert") pod "0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" (UID: "0e6ac0b5-0697-40d7-925c-68ff26e2f2bc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:40:58.032686 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.032660    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s" (OuterVolumeSpecName: "kube-api-access-7rl4s") pod "0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" (UID: "0e6ac0b5-0697-40d7-925c-68ff26e2f2bc"). InnerVolumeSpecName "kube-api-access-7rl4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:40:58.131912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.131833    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rl4s\" (UniqueName: \"kubernetes.io/projected/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-kube-api-access-7rl4s\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:40:58.131912 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.131863    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:40:58.136194 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.136160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-htq6x"]
Jul 09 09:40:58.138703 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:40:58.138679    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cb3d18_767c_44a4_b768_11c9acaa6545.slice/crio-985052d2e79902037a30fa4204b45b8b0f138bb3ee63b188bc5891e6e8cf6270 WatchSource:0}: Error finding container 985052d2e79902037a30fa4204b45b8b0f138bb3ee63b188bc5891e6e8cf6270: Status 404 returned error can't find the container with id 985052d2e79902037a30fa4204b45b8b0f138bb3ee63b188bc5891e6e8cf6270
Jul 09 09:40:58.515373 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.515338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x" event={"ID":"55cb3d18-767c-44a4-b768-11c9acaa6545","Type":"ContainerStarted","Data":"985052d2e79902037a30fa4204b45b8b0f138bb3ee63b188bc5891e6e8cf6270"}
Jul 09 09:40:58.516511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.516485    2576 generic.go:356] "Generic (PLEG): container finished" podID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" containerID="0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd" exitCode=0
Jul 09 09:40:58.516641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.516531    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b"
Jul 09 09:40:58.516641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.516546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" event={"ID":"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc","Type":"ContainerDied","Data":"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"}
Jul 09 09:40:58.516641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.516582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-vzq2b" event={"ID":"0e6ac0b5-0697-40d7-925c-68ff26e2f2bc","Type":"ContainerDied","Data":"e5cb9ab540d114467d7592498727a5ab081c982f830fea84efbef5d760ae8bc2"}
Jul 09 09:40:58.516641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.516603    2576 scope.go:117] "RemoveContainer" containerID="0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"
Jul 09 09:40:58.542841 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.542812    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:58.546775 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.546752    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5876b4f96d-vzq2b"]
Jul 09 09:40:58.568259 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.568243    2576 scope.go:117] "RemoveContainer" containerID="0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"
Jul 09 09:40:58.568551 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:40:58.568530    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd\": container with ID starting with 0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd not found: ID does not exist" containerID="0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"
Jul 09 09:40:58.568601 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:58.568559    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd"} err="failed to get container status \"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd\": rpc error: code = NotFound desc = could not find container \"0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd\": container with ID starting with 0fd32d2f9351a8328150633eb407a63f29d51a8ebcde6bc4e78da7b17eade2cd not found: ID does not exist"
Jul 09 09:40:59.522746 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:59.522692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x" event={"ID":"55cb3d18-767c-44a4-b768-11c9acaa6545","Type":"ContainerStarted","Data":"2b3dcfe39d6df05bbd2a07ee5e958cad60a39f2198cc38b5752471815b6034a5"}
Jul 09 09:40:59.523217 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:59.522808    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:40:59.540993 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:59.540955    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" path="/var/lib/kubelet/pods/0e6ac0b5-0697-40d7-925c-68ff26e2f2bc/volumes"
Jul 09 09:40:59.541630 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:40:59.541591    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x" podStartSLOduration=2.086150999 podStartE2EDuration="2.541575364s" totalImagesPullingTime="455.424365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:40:59.539845823 +0000 UTC m=+444.575655528" watchObservedRunningTime="2026-07-09 09:40:59.541575364 +0000 UTC m=+444.577385126"
Jul 09 09:41:14.815692 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:14.815657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:14.879292 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:14.879266    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:25.385078 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:25.385034    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:41:25.491960 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:25.491912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:25.545520 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:25.545481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:25.602232 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:25.602211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:30.532934 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:30.532903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5876b4f96d-htq6x"
Jul 09 09:41:31.488819 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.488749    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:31.593321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.593280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:31.681757 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.681725    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6g2nc"]
Jul 09 09:41:31.682069 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.682055    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" containerName="manager"
Jul 09 09:41:31.682115 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.682072    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" containerName="manager"
Jul 09 09:41:31.682149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.682131    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6ac0b5-0697-40d7-925c-68ff26e2f2bc" containerName="manager"
Jul 09 09:41:31.685119 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.685102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.689660 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.689639    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 09:41:31.690710 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.690689    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r9ljd\""
Jul 09 09:41:31.704783 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.704755    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6g2nc"]
Jul 09 09:41:31.779967 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.779908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:31.783994 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.783972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm79j\" (UniqueName: \"kubernetes.io/projected/34aa4be3-2a02-441a-90ea-a64af040e0de-kube-api-access-nm79j\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.784087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.784032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aa4be3-2a02-441a-90ea-a64af040e0de-cert\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.864105 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.864079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:31.885253 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.885220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aa4be3-2a02-441a-90ea-a64af040e0de-cert\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.885355 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.885319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm79j\" (UniqueName: \"kubernetes.io/projected/34aa4be3-2a02-441a-90ea-a64af040e0de-kube-api-access-nm79j\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.887627 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.887610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34aa4be3-2a02-441a-90ea-a64af040e0de-cert\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.896642 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.896623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm79j\" (UniqueName: \"kubernetes.io/projected/34aa4be3-2a02-441a-90ea-a64af040e0de-kube-api-access-nm79j\") pod \"odh-model-controller-84b54b7f97-6g2nc\" (UID: \"34aa4be3-2a02-441a-90ea-a64af040e0de\") " pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:31.995567 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:31.995536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:32.123996 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:32.123885    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6g2nc"]
Jul 09 09:41:32.126588 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:41:32.126563    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aa4be3_2a02_441a_90ea_a64af040e0de.slice/crio-97a715624c85077139681a44aa04b381778c52e2a8a4ff294c73f57c3a0ac192 WatchSource:0}: Error finding container 97a715624c85077139681a44aa04b381778c52e2a8a4ff294c73f57c3a0ac192: Status 404 returned error can't find the container with id 97a715624c85077139681a44aa04b381778c52e2a8a4ff294c73f57c3a0ac192
Jul 09 09:41:32.527821 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:32.527788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:32.607616 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:32.607581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:41:32.667287 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:32.667249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6g2nc" event={"ID":"34aa4be3-2a02-441a-90ea-a64af040e0de","Type":"ContainerStarted","Data":"97a715624c85077139681a44aa04b381778c52e2a8a4ff294c73f57c3a0ac192"}
Jul 09 09:41:34.677209 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:34.677157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6g2nc" event={"ID":"34aa4be3-2a02-441a-90ea-a64af040e0de","Type":"ContainerStarted","Data":"99d550cb04852ab1157e33a02d15f3ed1ecb26c908aefe4da8635eb38a4f9a9d"}
Jul 09 09:41:34.714144 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:34.714090    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-6g2nc" podStartSLOduration=1.352593475 podStartE2EDuration="3.714071962s" totalImagesPullingTime="2.361478487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:41:34.711759495 +0000 UTC m=+479.747569199" watchObservedRunningTime="2026-07-09 09:41:34.714071962 +0000 UTC m=+479.749881666"
Jul 09 09:41:35.681398 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:35.681322    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:46.687726 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:46.687693    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-6g2nc"
Jul 09 09:41:50.405914 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.405866    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5685ddcd95-lk7pm" podUID="90247838-f4d2-4016-94df-4ca9e93472cf" containerName="console" containerID="cri-o://ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76" gracePeriod=15
Jul 09 09:41:50.653599 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.653576    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:41:50.734955 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.734930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735067 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.734963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kczvx\" (UniqueName: \"kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735067 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735005    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735067 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735035    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735245 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735078    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735245 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735349 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert\") pod \"90247838-f4d2-4016-94df-4ca9e93472cf\" (UID: \"90247838-f4d2-4016-94df-4ca9e93472cf\") "
Jul 09 09:41:50.735404 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735388    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:41:50.735572 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735538    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config" (OuterVolumeSpecName: "console-config") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:41:50.735572 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735556    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-trusted-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.735572 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735559    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:41:50.735756 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.735672    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:41:50.737149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.737125    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx" (OuterVolumeSpecName: "kube-api-access-kczvx") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "kube-api-access-kczvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:41:50.737284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.737260    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:41:50.737373 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.737350    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "90247838-f4d2-4016-94df-4ca9e93472cf" (UID: "90247838-f4d2-4016-94df-4ca9e93472cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:41:50.744734 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.744700    2576 generic.go:356] "Generic (PLEG): container finished" podID="90247838-f4d2-4016-94df-4ca9e93472cf" containerID="ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76" exitCode=0
Jul 09 09:41:50.744838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.744742    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5685ddcd95-lk7pm"
Jul 09 09:41:50.744838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.744747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5685ddcd95-lk7pm" event={"ID":"90247838-f4d2-4016-94df-4ca9e93472cf","Type":"ContainerDied","Data":"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"}
Jul 09 09:41:50.744838 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.744826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5685ddcd95-lk7pm" event={"ID":"90247838-f4d2-4016-94df-4ca9e93472cf","Type":"ContainerDied","Data":"44b752928735bd3680ce50530feb546be087cdd0d22a020c7bd04a12b86868f4"}
Jul 09 09:41:50.744960 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.744851    2576 scope.go:117] "RemoveContainer" containerID="ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"
Jul 09 09:41:50.753619 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.753604    2576 scope.go:117] "RemoveContainer" containerID="ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"
Jul 09 09:41:50.753882 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:41:50.753856    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76\": container with ID starting with ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76 not found: ID does not exist" containerID="ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"
Jul 09 09:41:50.753937 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.753882    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76"} err="failed to get container status \"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76\": rpc error: code = NotFound desc = could not find container \"ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76\": container with ID starting with ed0c581685a1b4b311b19d3969b4c08ccb664ffd2bc2d8d9900ad0b577834d76 not found: ID does not exist"
Jul 09 09:41:50.771839 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.771815    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:41:50.775845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.775826    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5685ddcd95-lk7pm"]
Jul 09 09:41:50.836167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836143    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-console-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.836167 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836165    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-service-ca\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.836294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836175    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.836294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836206    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczvx\" (UniqueName: \"kubernetes.io/projected/90247838-f4d2-4016-94df-4ca9e93472cf-kube-api-access-kczvx\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.836294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836216    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90247838-f4d2-4016-94df-4ca9e93472cf-oauth-serving-cert\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:50.836294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:50.836224    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90247838-f4d2-4016-94df-4ca9e93472cf-console-oauth-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:41:51.541535 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:41:51.541500    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90247838-f4d2-4016-94df-4ca9e93472cf" path="/var/lib/kubelet/pods/90247838-f4d2-4016-94df-4ca9e93472cf/volumes"
Jul 09 09:42:25.490754 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:42:25.490715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:42:52.404598 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:42:52.404551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:06.123053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:06.123016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:06.220953 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:06.220924    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.237988 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.237948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.454616 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.454581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.548454 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.548383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.549310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.549289    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:43:18.549607 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.549596    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90247838-f4d2-4016-94df-4ca9e93472cf" containerName="console"
Jul 09 09:43:18.549655 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.549609    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90247838-f4d2-4016-94df-4ca9e93472cf" containerName="console"
Jul 09 09:43:18.549692 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.549668    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90247838-f4d2-4016-94df-4ca9e93472cf" containerName="console"
Jul 09 09:43:18.553033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.553013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.556308 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.556284    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1980c-kube-rbac-proxy-sar-config\""
Jul 09 09:43:18.556442 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.556365    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1980c-predictor-serving-cert\""
Jul 09 09:43:18.556523 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.556454    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tmt4d\""
Jul 09 09:43:18.557446 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.557424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 09:43:18.557623 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.557607    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 09:43:18.564159 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.564136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:43:18.625726 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.625698    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.703767 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.703738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.703953 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.703829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.703953 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.703877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgxg\" (UniqueName: \"kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.708325 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.708295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.779687 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.779656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.804881 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.804809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.804881 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.804860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.805058 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.804900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgxg\" (UniqueName: \"kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.805058 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:43:18.804974    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1980c-predictor-serving-cert: secret "error-404-isvc-1980c-predictor-serving-cert" not found
Jul 09 09:43:18.805058 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:43:18.805035    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls podName:cc6cecf0-dc4e-42a0-a988-d77b3bd974ee nodeName:}" failed. No retries permitted until 2026-07-09 09:43:19.305015957 +0000 UTC m=+584.340825646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls") pod "error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" (UID: "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee") : secret "error-404-isvc-1980c-predictor-serving-cert" not found
Jul 09 09:43:18.805564 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.805543    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.813401 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.813380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgxg\" (UniqueName: \"kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:18.880201 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.880147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:18.981394 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:18.981360    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:19.126714 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.126648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:19.219973 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.219949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:19.309086 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.309059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:19.311379 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.311355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") pod \"error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:19.466639 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.466616    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:19.596276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:19.596247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:43:19.599120 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:43:19.599096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6cecf0_dc4e_42a0_a988_d77b3bd974ee.slice/crio-a264ed28cd6c8aea30a3e2a9797bd1efccb537787d77fd3155a0567f9cb4ca1c WatchSource:0}: Error finding container a264ed28cd6c8aea30a3e2a9797bd1efccb537787d77fd3155a0567f9cb4ca1c: Status 404 returned error can't find the container with id a264ed28cd6c8aea30a3e2a9797bd1efccb537787d77fd3155a0567f9cb4ca1c
Jul 09 09:43:20.130952 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:20.130917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerStarted","Data":"a264ed28cd6c8aea30a3e2a9797bd1efccb537787d77fd3155a0567f9cb4ca1c"}
Jul 09 09:43:29.962815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:29.962774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:30.063330 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:30.063222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:33.202686 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:33.202655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerStarted","Data":"799cfc52eee742eb84b4c858156b26dccb28594b923c5c3d9f9648351662fe51"}
Jul 09 09:43:35.213781 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:35.213749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerStarted","Data":"f079321185fb24e704d2697d67c032043010432da0648e0324b8c699a231a919"}
Jul 09 09:43:35.214151 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:35.213889    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:35.237429 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:35.237377    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podStartSLOduration=2.053400084 podStartE2EDuration="17.237357308s" totalImagesPullingTime="15.183957224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:35.234403866 +0000 UTC m=+600.270213597" watchObservedRunningTime="2026-07-09 09:43:35.237357308 +0000 UTC m=+600.273167016"
Jul 09 09:43:36.218380 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:36.218348    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:36.219663 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:36.219639    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:43:37.222276 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:37.222230    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:43:42.226722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:42.226692    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:43:42.227171 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:42.227145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:43:51.985141 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:51.985107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:43:52.285033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:52.227367    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:43:53.579896 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:43:53.579859    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:02.227963 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:02.227866    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:44:12.227722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:12.227679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 09:44:22.227934 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:22.227903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:44:38.243869 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.243833    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:44:38.246035 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.246015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.248510 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.248486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1980c-kube-rbac-proxy-sar-config\""
Jul 09 09:44:38.248729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.248489    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1980c-serving-cert\""
Jul 09 09:44:38.248729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.248668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:38.255312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.255287    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:44:38.431221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.431176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.431221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.431229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.532156 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.532089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.532156 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.532121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.532359 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:44:38.532289    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-1980c-serving-cert: secret "switch-graph-1980c-serving-cert" not found
Jul 09 09:44:38.532359 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:44:38.532345    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls podName:ec8a48ba-77aa-4691-9b6c-e0b99829569a nodeName:}" failed. No retries permitted until 2026-07-09 09:44:39.032331272 +0000 UTC m=+664.068140955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls") pod "switch-graph-1980c-84fdbd9b6f-d79c9" (UID: "ec8a48ba-77aa-4691-9b6c-e0b99829569a") : secret "switch-graph-1980c-serving-cert" not found
Jul 09 09:44:38.532781 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.532754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:38.612629 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:38.612597    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:39.034221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.034175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:39.036294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.036270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") pod \"switch-graph-1980c-84fdbd9b6f-d79c9\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") " pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:39.156639 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.156611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:39.279905 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.279847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:44:39.282060 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:44:39.282033    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8a48ba_77aa_4691_9b6c_e0b99829569a.slice/crio-eb9bca6803ee826721ba56777b8f60c66d147ac8afd6e32d42c0af21ace740ee WatchSource:0}: Error finding container eb9bca6803ee826721ba56777b8f60c66d147ac8afd6e32d42c0af21ace740ee: Status 404 returned error can't find the container with id eb9bca6803ee826721ba56777b8f60c66d147ac8afd6e32d42c0af21ace740ee
Jul 09 09:44:39.283987 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.283970    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:44:39.482359 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:39.482332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" event={"ID":"ec8a48ba-77aa-4691-9b6c-e0b99829569a","Type":"ContainerStarted","Data":"eb9bca6803ee826721ba56777b8f60c66d147ac8afd6e32d42c0af21ace740ee"}
Jul 09 09:44:42.498376 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:42.498326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" event={"ID":"ec8a48ba-77aa-4691-9b6c-e0b99829569a","Type":"ContainerStarted","Data":"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"}
Jul 09 09:44:42.498821 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:42.498422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:42.520343 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:42.520286    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podStartSLOduration=2.241564547 podStartE2EDuration="4.520265788s" totalImagesPullingTime="2.278701241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:44:42.519721316 +0000 UTC m=+667.555531022" watchObservedRunningTime="2026-07-09 09:44:42.520265788 +0000 UTC m=+667.556075496"
Jul 09 09:44:48.508278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:48.508245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:44:52.371986 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.371954    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:44:52.372423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.372140    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" containerID="cri-o://87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46" gracePeriod=30
Jul 09 09:44:52.405745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.405708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:52.545685 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.545630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:52.621123 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.621076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:52.627575 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.627511    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:44:52.627763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.627734    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container" containerID="cri-o://799cfc52eee742eb84b4c858156b26dccb28594b923c5c3d9f9648351662fe51" gracePeriod=30
Jul 09 09:44:52.627889 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.627767    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kube-rbac-proxy" containerID="cri-o://f079321185fb24e704d2697d67c032043010432da0648e0324b8c699a231a919" gracePeriod=30
Jul 09 09:44:52.649596 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.649574    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:44:52.653476 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.652492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.655485 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.655461    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d1ee-predictor-serving-cert\""
Jul 09 09:44:52.655795 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.655772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d1ee-kube-rbac-proxy-sar-config\""
Jul 09 09:44:52.662406 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.662386    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:44:52.680239 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.680216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:52.730999 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.730973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67prb\" (UniqueName: \"kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.731121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.731020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.731224 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.731157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.832296 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.832265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.832408 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.832311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67prb\" (UniqueName: \"kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.832408 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.832348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.832996 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.832972    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.834623 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.834604    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.839854 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.839829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67prb\" (UniqueName: \"kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb\") pod \"error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") " pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:52.850070 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.850047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:52.966652 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:52.966629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:53.077652 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.077613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:44:53.095491 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.095467    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:44:53.097740 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:44:53.097707    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74da4a0_f711_45a7_b0dc_db566880a5c2.slice/crio-3ad690b7b3cf62b134965a309a29bc85ae0e8307e8f110ff4f360387b23278f3 WatchSource:0}: Error finding container 3ad690b7b3cf62b134965a309a29bc85ae0e8307e8f110ff4f360387b23278f3: Status 404 returned error can't find the container with id 3ad690b7b3cf62b134965a309a29bc85ae0e8307e8f110ff4f360387b23278f3
Jul 09 09:44:53.507109 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.507075    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:44:53.546981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.546954    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerID="f079321185fb24e704d2697d67c032043010432da0648e0324b8c699a231a919" exitCode=2
Jul 09 09:44:53.547131 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.547020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerDied","Data":"f079321185fb24e704d2697d67c032043010432da0648e0324b8c699a231a919"}
Jul 09 09:44:53.548765 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.548737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerStarted","Data":"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"}
Jul 09 09:44:53.548871 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.548793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerStarted","Data":"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"}
Jul 09 09:44:53.548871 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.548823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerStarted","Data":"3ad690b7b3cf62b134965a309a29bc85ae0e8307e8f110ff4f360387b23278f3"}
Jul 09 09:44:53.548940 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.548895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:53.568697 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:53.568652    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podStartSLOduration=1.56864134 podStartE2EDuration="1.56864134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:44:53.566404619 +0000 UTC m=+678.602214325" watchObservedRunningTime="2026-07-09 09:44:53.56864134 +0000 UTC m=+678.604451045"
Jul 09 09:44:54.553087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:54.553054    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:44:54.554365 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:54.554339    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:44:55.559057 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.559031    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerID="799cfc52eee742eb84b4c858156b26dccb28594b923c5c3d9f9648351662fe51" exitCode=0
Jul 09 09:44:55.559421 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.559098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerDied","Data":"799cfc52eee742eb84b4c858156b26dccb28594b923c5c3d9f9648351662fe51"}
Jul 09 09:44:55.559470 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.559451    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:44:55.584596 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.584576    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:44:55.650092 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.650026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config\") pod \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") "
Jul 09 09:44:55.650222 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.650092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") pod \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") "
Jul 09 09:44:55.650222 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.650193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlgxg\" (UniqueName: \"kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg\") pod \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\" (UID: \"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee\") "
Jul 09 09:44:55.650351 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.650337    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" (UID: "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:44:55.650493 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.650473    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:44:55.652225 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.652203    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" (UID: "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:44:55.652312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.652267    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg" (OuterVolumeSpecName: "kube-api-access-vlgxg") pod "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" (UID: "cc6cecf0-dc4e-42a0-a988-d77b3bd974ee"). InnerVolumeSpecName "kube-api-access-vlgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:44:55.751135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.751109    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:44:55.751135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:55.751130    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgxg\" (UniqueName: \"kubernetes.io/projected/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee-kube-api-access-vlgxg\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:44:56.565320 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.565286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf" event={"ID":"cc6cecf0-dc4e-42a0-a988-d77b3bd974ee","Type":"ContainerDied","Data":"a264ed28cd6c8aea30a3e2a9797bd1efccb537787d77fd3155a0567f9cb4ca1c"}
Jul 09 09:44:56.565736 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.565327    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"
Jul 09 09:44:56.565736 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.565352    2576 scope.go:117] "RemoveContainer" containerID="f079321185fb24e704d2697d67c032043010432da0648e0324b8c699a231a919"
Jul 09 09:44:56.573774 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.573756    2576 scope.go:117] "RemoveContainer" containerID="799cfc52eee742eb84b4c858156b26dccb28594b923c5c3d9f9648351662fe51"
Jul 09 09:44:56.591790 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.591769    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:44:56.595941 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:56.595920    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1980c-predictor-5ff6dfb79c-x64tf"]
Jul 09 09:44:57.541447 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:57.541413    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" path="/var/lib/kubelet/pods/cc6cecf0-dc4e-42a0-a988-d77b3bd974ee/volumes"
Jul 09 09:44:58.506877 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:44:58.506840    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:00.564060 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:00.564032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:45:00.564540 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:00.564515    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:45:03.506908 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:03.506863    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:03.507317 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:03.506975    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:45:08.506730 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:08.506689    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:10.565194 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:10.565154    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:45:13.507475 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:13.507434    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:18.506837 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:18.506798    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:20.565081 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:20.565043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:45:22.560143 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.560120    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:45:22.666366 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.666286    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") pod \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") "
Jul 09 09:45:22.666366 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.666325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle\") pod \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\" (UID: \"ec8a48ba-77aa-4691-9b6c-e0b99829569a\") "
Jul 09 09:45:22.666735 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.666710    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ec8a48ba-77aa-4691-9b6c-e0b99829569a" (UID: "ec8a48ba-77aa-4691-9b6c-e0b99829569a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:45:22.668336 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.668310    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec8a48ba-77aa-4691-9b6c-e0b99829569a" (UID: "ec8a48ba-77aa-4691-9b6c-e0b99829569a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:45:22.676613 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.676591    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerID="87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46" exitCode=137
Jul 09 09:45:22.676710 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.676628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" event={"ID":"ec8a48ba-77aa-4691-9b6c-e0b99829569a","Type":"ContainerDied","Data":"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"}
Jul 09 09:45:22.676710 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.676636    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"
Jul 09 09:45:22.676710 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.676693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9" event={"ID":"ec8a48ba-77aa-4691-9b6c-e0b99829569a","Type":"ContainerDied","Data":"eb9bca6803ee826721ba56777b8f60c66d147ac8afd6e32d42c0af21ace740ee"}
Jul 09 09:45:22.676829 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.676714    2576 scope.go:117] "RemoveContainer" containerID="87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"
Jul 09 09:45:22.685436 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.685398    2576 scope.go:117] "RemoveContainer" containerID="87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"
Jul 09 09:45:22.685765 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:45:22.685746    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46\": container with ID starting with 87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46 not found: ID does not exist" containerID="87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"
Jul 09 09:45:22.685819 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.685774    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46"} err="failed to get container status \"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46\": rpc error: code = NotFound desc = could not find container \"87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46\": container with ID starting with 87c7d013d03fdcd21ae2977afde4685321a90b762c65329a8bab746f810c3a46 not found: ID does not exist"
Jul 09 09:45:22.710740 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.710717    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:45:22.715134 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.715113    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1980c-84fdbd9b6f-d79c9"]
Jul 09 09:45:22.767645 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.767623    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8a48ba-77aa-4691-9b6c-e0b99829569a-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:45:22.767645 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:22.767646    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec8a48ba-77aa-4691-9b6c-e0b99829569a-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:45:23.540684 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:23.540649    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" path="/var/lib/kubelet/pods/ec8a48ba-77aa-4691-9b6c-e0b99829569a/volumes"
Jul 09 09:45:30.565509 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:30.565432    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 09:45:38.225818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.225779    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226136    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226149    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226173    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kube-rbac-proxy"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226194    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kube-rbac-proxy"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226201    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226207    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c"
Jul 09 09:45:38.226248 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226252    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8a48ba-77aa-4691-9b6c-e0b99829569a" containerName="switch-graph-1980c"
Jul 09 09:45:38.226467 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226260    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kserve-container"
Jul 09 09:45:38.226467 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.226267    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6cecf0-dc4e-42a0-a988-d77b3bd974ee" containerName="kube-rbac-proxy"
Jul 09 09:45:38.230427 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.230407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.232837 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.232816    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 09 09:45:38.232837 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.232827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 09 09:45:38.237976 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.237954    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:45:38.238278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.238255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:38.395453 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.395424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.395555 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.395460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.496469 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.496412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.496469 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.496448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.496588 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:45:38.496560    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 09 09:45:38.496635 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:45:38.496624    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls podName:e70daffd-8b6b-4248-9580-d442fb7fe972 nodeName:}" failed. No retries permitted until 2026-07-09 09:45:38.996607684 +0000 UTC m=+724.032417366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls") pod "model-chainer-745f4f776f-jldbm" (UID: "e70daffd-8b6b-4248-9580-d442fb7fe972") : secret "model-chainer-serving-cert" not found
Jul 09 09:45:38.496981 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.496965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:38.723628 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:38.723575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:39.000703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.000659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:39.002984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.002966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") pod \"model-chainer-745f4f776f-jldbm\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") " pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:39.141664 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.141635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:39.268000 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.267976    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:45:39.269858 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:45:39.269826    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70daffd_8b6b_4248_9580_d442fb7fe972.slice/crio-2f728f85a098a2c494ce2bd6bea29ede7c372f18b9bc6efe59932cbcb8122769 WatchSource:0}: Error finding container 2f728f85a098a2c494ce2bd6bea29ede7c372f18b9bc6efe59932cbcb8122769: Status 404 returned error can't find the container with id 2f728f85a098a2c494ce2bd6bea29ede7c372f18b9bc6efe59932cbcb8122769
Jul 09 09:45:39.749738 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.749706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" event={"ID":"e70daffd-8b6b-4248-9580-d442fb7fe972","Type":"ContainerStarted","Data":"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"}
Jul 09 09:45:39.749876 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.749753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" event={"ID":"e70daffd-8b6b-4248-9580-d442fb7fe972","Type":"ContainerStarted","Data":"2f728f85a098a2c494ce2bd6bea29ede7c372f18b9bc6efe59932cbcb8122769"}
Jul 09 09:45:39.749876 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.749783    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:39.768638 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:39.768590    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podStartSLOduration=1.7685756769999998 podStartE2EDuration="1.768575677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:45:39.76707637 +0000 UTC m=+724.802886074" watchObservedRunningTime="2026-07-09 09:45:39.768575677 +0000 UTC m=+724.804385382"
Jul 09 09:45:40.565455 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:40.565425    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:45:45.760192 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:45.760160    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:45:48.321683 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.321648    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:45:48.322145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.321885    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" containerID="cri-o://41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48" gracePeriod=30
Jul 09 09:45:48.342631 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.342605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.415684 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.415654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.515120 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.513608    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.625745 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.625677    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:45:48.629325 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.629303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.631698 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.631676    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8b307-predictor-serving-cert\""
Jul 09 09:45:48.631814 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.631695    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8b307-kube-rbac-proxy-sar-config\""
Jul 09 09:45:48.635052 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.635033    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.638779 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.638754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:45:48.756760 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.756727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.778818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.778787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.778940 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.778835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.778940 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.778919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtq9\" (UniqueName: \"kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.861582 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.861552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:48.879903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.879841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.879903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.879878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.879903 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.879902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtq9\" (UniqueName: \"kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.880534 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.880513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.882205 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.882173    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.889094 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.889072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtq9\" (UniqueName: \"kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9\") pod \"error-404-isvc-8b307-predictor-59c78cddf5-l7zxp\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") " pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.940897 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.940867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:48.964066 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:48.964037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:49.064656 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.064630    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:45:49.067330 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:45:49.067300    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5d66b0_d347_4b61_bbf0_e000397703be.slice/crio-6b0ed7d6f6418c1cc01a7a2e7496d9efd4c0db2f9925e22306661a55aa85270d WatchSource:0}: Error finding container 6b0ed7d6f6418c1cc01a7a2e7496d9efd4c0db2f9925e22306661a55aa85270d: Status 404 returned error can't find the container with id 6b0ed7d6f6418c1cc01a7a2e7496d9efd4c0db2f9925e22306661a55aa85270d
Jul 09 09:45:49.799195 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.799154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerStarted","Data":"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"}
Jul 09 09:45:49.799693 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.799227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerStarted","Data":"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"}
Jul 09 09:45:49.799693 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.799256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerStarted","Data":"6b0ed7d6f6418c1cc01a7a2e7496d9efd4c0db2f9925e22306661a55aa85270d"}
Jul 09 09:45:49.799693 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.799322    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:49.819709 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:49.819663    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podStartSLOduration=1.8196507 podStartE2EDuration="1.8196507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:45:49.818301601 +0000 UTC m=+734.854111306" watchObservedRunningTime="2026-07-09 09:45:49.8196507 +0000 UTC m=+734.855460406"
Jul 09 09:45:50.757898 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:50.757862    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:50.803766 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:50.803728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:50.804889 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:50.804862    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:45:51.809014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:51.808977    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:45:52.612033 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.611999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:45:52.615168 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.615150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.617459 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.617440    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8d1ee-serving-cert\""
Jul 09 09:45:52.617704 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.617687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8d1ee-kube-rbac-proxy-sar-config\""
Jul 09 09:45:52.623829 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.623799    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:45:52.626084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.626062    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:52.707376 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.707345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.707485 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.707384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.807754 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.807730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.807853 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.807767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.808389 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.808365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.810198 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.810162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls\") pod \"switch-graph-8d1ee-7c6544c954-fv9qq\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") " pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:52.839841 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.839818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:45:52.926655 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:52.926588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:53.052294 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:53.052259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:45:53.055853 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:45:53.055828    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda655b312_6a82_4ba3_8a8a_9465580aab2f.slice/crio-284289480b1e8114fad7b34a41a92d6316744b1a20b5fa5a8f412a8772630caf WatchSource:0}: Error finding container 284289480b1e8114fad7b34a41a92d6316744b1a20b5fa5a8f412a8772630caf: Status 404 returned error can't find the container with id 284289480b1e8114fad7b34a41a92d6316744b1a20b5fa5a8f412a8772630caf
Jul 09 09:45:53.819250 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:53.819210    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" event={"ID":"a655b312-6a82-4ba3-8a8a-9465580aab2f","Type":"ContainerStarted","Data":"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"}
Jul 09 09:45:53.819637 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:53.819260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" event={"ID":"a655b312-6a82-4ba3-8a8a-9465580aab2f","Type":"ContainerStarted","Data":"284289480b1e8114fad7b34a41a92d6316744b1a20b5fa5a8f412a8772630caf"}
Jul 09 09:45:53.819637 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:53.819343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:45:53.837762 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:53.837719    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podStartSLOduration=1.837705137 podStartE2EDuration="1.837705137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:45:53.836151634 +0000 UTC m=+738.871961338" watchObservedRunningTime="2026-07-09 09:45:53.837705137 +0000 UTC m=+738.873514842"
Jul 09 09:45:55.758697 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:55.758654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:45:56.813897 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:56.813864    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:45:56.814387 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:56.814358    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:45:59.829742 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:45:59.829709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:46:00.758950 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:00.758912    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:46:00.759135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:00.759026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:46:05.757999 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:05.757962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:46:06.814363 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:06.814325    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:46:10.757958 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:10.757913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:46:15.758768 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:15.758731    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:46:16.815145 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:16.815107    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:46:18.468840 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.468809    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:46:18.511394 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.511368    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") pod \"e70daffd-8b6b-4248-9580-d442fb7fe972\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") "
Jul 09 09:46:18.511504 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.511447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle\") pod \"e70daffd-8b6b-4248-9580-d442fb7fe972\" (UID: \"e70daffd-8b6b-4248-9580-d442fb7fe972\") "
Jul 09 09:46:18.511799 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.511775    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e70daffd-8b6b-4248-9580-d442fb7fe972" (UID: "e70daffd-8b6b-4248-9580-d442fb7fe972"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:46:18.513332 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.513314    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e70daffd-8b6b-4248-9580-d442fb7fe972" (UID: "e70daffd-8b6b-4248-9580-d442fb7fe972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:46:18.612668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.612587    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70daffd-8b6b-4248-9580-d442fb7fe972-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:46:18.612668 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.612622    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e70daffd-8b6b-4248-9580-d442fb7fe972-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:46:18.927818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.927729    2576 generic.go:356] "Generic (PLEG): container finished" podID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerID="41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48" exitCode=0
Jul 09 09:46:18.927818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.927774    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"
Jul 09 09:46:18.927818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.927797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" event={"ID":"e70daffd-8b6b-4248-9580-d442fb7fe972","Type":"ContainerDied","Data":"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"}
Jul 09 09:46:18.928053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.927850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm" event={"ID":"e70daffd-8b6b-4248-9580-d442fb7fe972","Type":"ContainerDied","Data":"2f728f85a098a2c494ce2bd6bea29ede7c372f18b9bc6efe59932cbcb8122769"}
Jul 09 09:46:18.928053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.927882    2576 scope.go:117] "RemoveContainer" containerID="41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"
Jul 09 09:46:18.937613 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.937594    2576 scope.go:117] "RemoveContainer" containerID="41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"
Jul 09 09:46:18.937843 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:46:18.937826    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48\": container with ID starting with 41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48 not found: ID does not exist" containerID="41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"
Jul 09 09:46:18.937890 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.937850    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48"} err="failed to get container status \"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48\": rpc error: code = NotFound desc = could not find container \"41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48\": container with ID starting with 41992e777481daf4361bbee8f30d1a953db261d0bc28176eb1cec95187550b48 not found: ID does not exist"
Jul 09 09:46:18.947787 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.947764    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:46:18.952496 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:18.952476    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-745f4f776f-jldbm"]
Jul 09 09:46:19.541144 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:19.541113    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" path="/var/lib/kubelet/pods/e70daffd-8b6b-4248-9580-d442fb7fe972/volumes"
Jul 09 09:46:26.814948 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:26.814913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 09:46:36.815353 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:36.815320    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:46:48.547630 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.547594    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:46:48.548088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.548068    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer"
Jul 09 09:46:48.548088 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.548088    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer"
Jul 09 09:46:48.548230 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.548215    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70daffd-8b6b-4248-9580-d442fb7fe972" containerName="model-chainer"
Jul 09 09:46:48.551546 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.551519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:46:48.552211 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.552192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.554879 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.554862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8b307-kube-rbac-proxy-sar-config\""
Jul 09 09:46:48.555624 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.555602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8b307-serving-cert\""
Jul 09 09:46:48.560659 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.560638    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:46:48.648501 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.648476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.648606 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.648504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.740315 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.740273    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:46:48.749454 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.749434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.749529 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.749464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.750100 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.750084    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.751740 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.751723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls\") pod \"sequence-graph-8b307-86775f9586-b64sl\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") " pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.863494 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.863445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:48.991463 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:48.991440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:46:48.993984 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:46:48.993961    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4412a1d7_206b_4ee4_ac75_5368eeb73be8.slice/crio-0485fc9339ed300a307ebe735cf083d0768950514a440a1a7faf0d717ab05f2e WatchSource:0}: Error finding container 0485fc9339ed300a307ebe735cf083d0768950514a440a1a7faf0d717ab05f2e: Status 404 returned error can't find the container with id 0485fc9339ed300a307ebe735cf083d0768950514a440a1a7faf0d717ab05f2e
Jul 09 09:46:49.062295 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:49.062269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" event={"ID":"4412a1d7-206b-4ee4-ac75-5368eeb73be8","Type":"ContainerStarted","Data":"7cf6247da285afc79d0337a1e1baa919d302c8396893851936c6b34d122b2415"}
Jul 09 09:46:49.062413 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:49.062316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" event={"ID":"4412a1d7-206b-4ee4-ac75-5368eeb73be8","Type":"ContainerStarted","Data":"0485fc9339ed300a307ebe735cf083d0768950514a440a1a7faf0d717ab05f2e"}
Jul 09 09:46:49.062413 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:49.062403    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:46:49.079724 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:49.079683    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podStartSLOduration=1.079669217 podStartE2EDuration="1.079669217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:46:49.079047039 +0000 UTC m=+794.114856756" watchObservedRunningTime="2026-07-09 09:46:49.079669217 +0000 UTC m=+794.115478931"
Jul 09 09:46:55.073321 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:46:55.073291    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:49:47.754543 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:49:47.754493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:49:50.115584 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:49:50.115540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:49:50.191364 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:49:50.191331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:50:09.802001 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:50:09.801910    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:51:08.397441 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:51:08.397405    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.202551 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.202518    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:54:07.203139 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.202763    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" containerID="cri-o://7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761" gracePeriod=30
Jul 09 09:54:07.265084 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.265054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.370824 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.370784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.456031 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.455950    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.464213 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.463328    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:54:07.464213 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.463679    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container" containerID="cri-o://9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9" gracePeriod=30
Jul 09 09:54:07.464213 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.464094    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kube-rbac-proxy" containerID="cri-o://4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd" gracePeriod=30
Jul 09 09:54:07.491852 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.491825    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:54:07.495008 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.494990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.497827 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.497806    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b04db-predictor-serving-cert\""
Jul 09 09:54:07.497919 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.497845    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b04db-kube-rbac-proxy-sar-config\""
Jul 09 09:54:07.504206 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.504170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:54:07.525207 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.525165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.611747 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.611724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.611860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.611764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxshz\" (UniqueName: \"kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.611860 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.611797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.646951 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.646925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:07.712851 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.712793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.712851 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.712835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxshz\" (UniqueName: \"kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.713013 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.712859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.713531 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.713511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.715188 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.715157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.720328 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.720305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxshz\" (UniqueName: \"kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz\") pod \"error-404-isvc-b04db-predictor-f486cdf95-s8fmc\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") " pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.805529 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.805488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:07.938392 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:07.938368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:54:07.940270 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:54:07.940238    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1145536a_0c40_4de2_8755_d80cda6b21ba.slice/crio-e887a2a3973a696badb58822204f52ad9b00a2fae419d93206605afc0ee800b4 WatchSource:0}: Error finding container e887a2a3973a696badb58822204f52ad9b00a2fae419d93206605afc0ee800b4: Status 404 returned error can't find the container with id e887a2a3973a696badb58822204f52ad9b00a2fae419d93206605afc0ee800b4
Jul 09 09:54:08.011628 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:08.011607    2576 generic.go:356] "Generic (PLEG): container finished" podID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerID="4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd" exitCode=2
Jul 09 09:54:08.011729 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:08.011681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerDied","Data":"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"}
Jul 09 09:54:08.013189 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:08.013154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerStarted","Data":"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"}
Jul 09 09:54:08.013279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:08.013218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerStarted","Data":"e887a2a3973a696badb58822204f52ad9b00a2fae419d93206605afc0ee800b4"}
Jul 09 09:54:08.178845 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:08.178807    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:54:09.019889 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:09.019853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerStarted","Data":"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"}
Jul 09 09:54:09.020312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:09.019947    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:09.039379 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:09.039319    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podStartSLOduration=2.039299126 podStartE2EDuration="2.039299126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:09.037366368 +0000 UTC m=+1234.073176075" watchObservedRunningTime="2026-07-09 09:54:09.039299126 +0000 UTC m=+1234.075108831"
Jul 09 09:54:09.828043 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:09.827998    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:10.024798 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.024767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:10.026147 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.026118    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:10.519384 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.519356    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:54:10.640984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.640915    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config\") pod \"e74da4a0-f711-45a7-b0dc-db566880a5c2\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") "
Jul 09 09:54:10.640984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.640952    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls\") pod \"e74da4a0-f711-45a7-b0dc-db566880a5c2\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") "
Jul 09 09:54:10.641194 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.640987    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67prb\" (UniqueName: \"kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb\") pod \"e74da4a0-f711-45a7-b0dc-db566880a5c2\" (UID: \"e74da4a0-f711-45a7-b0dc-db566880a5c2\") "
Jul 09 09:54:10.641383 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.641358    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e74da4a0-f711-45a7-b0dc-db566880a5c2" (UID: "e74da4a0-f711-45a7-b0dc-db566880a5c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:54:10.643095 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.643066    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e74da4a0-f711-45a7-b0dc-db566880a5c2" (UID: "e74da4a0-f711-45a7-b0dc-db566880a5c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:54:10.643231 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.643159    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb" (OuterVolumeSpecName: "kube-api-access-67prb") pod "e74da4a0-f711-45a7-b0dc-db566880a5c2" (UID: "e74da4a0-f711-45a7-b0dc-db566880a5c2"). InnerVolumeSpecName "kube-api-access-67prb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:54:10.744666 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.742064    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:54:10.744666 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.742142    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74da4a0-f711-45a7-b0dc-db566880a5c2-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:54:10.744666 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:10.742162    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67prb\" (UniqueName: \"kubernetes.io/projected/e74da4a0-f711-45a7-b0dc-db566880a5c2-kube-api-access-67prb\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:54:11.030804 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.030763    2576 generic.go:356] "Generic (PLEG): container finished" podID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerID="9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9" exitCode=0
Jul 09 09:54:11.031319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.030822    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"
Jul 09 09:54:11.031319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.030848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerDied","Data":"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"}
Jul 09 09:54:11.031319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.030931    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6" event={"ID":"e74da4a0-f711-45a7-b0dc-db566880a5c2","Type":"ContainerDied","Data":"3ad690b7b3cf62b134965a309a29bc85ae0e8307e8f110ff4f360387b23278f3"}
Jul 09 09:54:11.031319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.030960    2576 scope.go:117] "RemoveContainer" containerID="4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"
Jul 09 09:54:11.031550 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.031404    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:11.039885 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.039868    2576 scope.go:117] "RemoveContainer" containerID="9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"
Jul 09 09:54:11.047890 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.047872    2576 scope.go:117] "RemoveContainer" containerID="4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"
Jul 09 09:54:11.048149 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:54:11.048127    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd\": container with ID starting with 4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd not found: ID does not exist" containerID="4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"
Jul 09 09:54:11.048221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.048158    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd"} err="failed to get container status \"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd\": rpc error: code = NotFound desc = could not find container \"4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd\": container with ID starting with 4d53c8b9125976a8365a5346867790bb9285668254fd29a92b393fe380bc44cd not found: ID does not exist"
Jul 09 09:54:11.048221 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.048192    2576 scope.go:117] "RemoveContainer" containerID="9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"
Jul 09 09:54:11.048458 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:54:11.048440    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9\": container with ID starting with 9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9 not found: ID does not exist" containerID="9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"
Jul 09 09:54:11.048516 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.048469    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9"} err="failed to get container status \"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9\": rpc error: code = NotFound desc = could not find container \"9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9\": container with ID starting with 9324ba4fe463242d19c0225f147be123f0307743263270fb64fe0c53aa66d0a9 not found: ID does not exist"
Jul 09 09:54:11.053019 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.052996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:54:11.057119 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.057096    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d1ee-predictor-5565fb77bf-qzfp6"]
Jul 09 09:54:11.542906 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:11.542873    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" path="/var/lib/kubelet/pods/e74da4a0-f711-45a7-b0dc-db566880a5c2/volumes"
Jul 09 09:54:14.828292 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:14.828250    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:16.037384 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:16.037353    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:54:16.037854 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:16.037825    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:19.827423 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:19.827380    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:19.827862 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:19.827493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:54:24.827090 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:24.827053    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:26.038755 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:26.038715    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:29.827351 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:29.827306    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:34.827349 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:34.827316    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:54:36.038076 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:36.038038    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:37.350534 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.350507    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:54:37.457502 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.457426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle\") pod \"a655b312-6a82-4ba3-8a8a-9465580aab2f\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") "
Jul 09 09:54:37.457502 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.457469    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls\") pod \"a655b312-6a82-4ba3-8a8a-9465580aab2f\" (UID: \"a655b312-6a82-4ba3-8a8a-9465580aab2f\") "
Jul 09 09:54:37.457777 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.457754    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a655b312-6a82-4ba3-8a8a-9465580aab2f" (UID: "a655b312-6a82-4ba3-8a8a-9465580aab2f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:54:37.459473 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.459451    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a655b312-6a82-4ba3-8a8a-9465580aab2f" (UID: "a655b312-6a82-4ba3-8a8a-9465580aab2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:54:37.558119 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.558091    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a655b312-6a82-4ba3-8a8a-9465580aab2f-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:54:37.558256 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:37.558124    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a655b312-6a82-4ba3-8a8a-9465580aab2f-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:54:38.156411 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.156375    2576 generic.go:356] "Generic (PLEG): container finished" podID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerID="7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761" exitCode=0
Jul 09 09:54:38.156576 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.156417    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"
Jul 09 09:54:38.156576 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.156464    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" event={"ID":"a655b312-6a82-4ba3-8a8a-9465580aab2f","Type":"ContainerDied","Data":"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"}
Jul 09 09:54:38.156576 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.156519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq" event={"ID":"a655b312-6a82-4ba3-8a8a-9465580aab2f","Type":"ContainerDied","Data":"284289480b1e8114fad7b34a41a92d6316744b1a20b5fa5a8f412a8772630caf"}
Jul 09 09:54:38.156576 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.156551    2576 scope.go:117] "RemoveContainer" containerID="7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"
Jul 09 09:54:38.165162 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.165146    2576 scope.go:117] "RemoveContainer" containerID="7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"
Jul 09 09:54:38.165454 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:54:38.165436    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761\": container with ID starting with 7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761 not found: ID does not exist" containerID="7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"
Jul 09 09:54:38.165519 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.165461    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761"} err="failed to get container status \"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761\": rpc error: code = NotFound desc = could not find container \"7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761\": container with ID starting with 7289da71c1fd552357be1474e33e79ea3d49408072c01d623937196f8f6a1761 not found: ID does not exist"
Jul 09 09:54:38.175031 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.175006    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:54:38.178564 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:38.178543    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d1ee-7c6544c954-fv9qq"]
Jul 09 09:54:39.541374 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:39.541341    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" path="/var/lib/kubelet/pods/a655b312-6a82-4ba3-8a8a-9465580aab2f/volumes"
Jul 09 09:54:46.038388 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:46.038349    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 09:54:56.039039 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:54:56.039010    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:55:03.133667 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.133628    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:55:03.134320 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.133832    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" containerID="cri-o://7cf6247da285afc79d0337a1e1baa919d302c8396893851936c6b34d122b2415" gracePeriod=30
Jul 09 09:55:03.175333 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.175299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:03.296030 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.296002    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:55:03.296278 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.296247    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container" containerID="cri-o://67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4" gracePeriod=30
Jul 09 09:55:03.296428 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.296317    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kube-rbac-proxy" containerID="cri-o://f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a" gracePeriod=30
Jul 09 09:55:03.420373 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420348    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:55:03.420714 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420692    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee"
Jul 09 09:55:03.420714 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420714    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420730    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420736    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420749    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kube-rbac-proxy"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420757    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kube-rbac-proxy"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420843    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kube-rbac-proxy"
Jul 09 09:55:03.420855 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420855    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a655b312-6a82-4ba3-8a8a-9465580aab2f" containerName="switch-graph-8d1ee"
Jul 09 09:55:03.421099 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.420863    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74da4a0-f711-45a7-b0dc-db566880a5c2" containerName="kserve-container"
Jul 09 09:55:03.423907 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.423892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.426640 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.426621    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3d13-predictor-serving-cert\""
Jul 09 09:55:03.426720 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.426644    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3d13-kube-rbac-proxy-sar-config\""
Jul 09 09:55:03.432230 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.432207    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:55:03.570984 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.570951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.571146 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.570996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnk5z\" (UniqueName: \"kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.571146 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.571064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.671983 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.671897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.671983 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.671970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.672212 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.671996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnk5z\" (UniqueName: \"kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.672635 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.672611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.674395 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.674376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.682887 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.682861    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnk5z\" (UniqueName: \"kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z\") pod \"error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.734931 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.734889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:03.874425 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:03.874382    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:55:03.878274 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:55:03.878250    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8634d0_e555_4d48_bccd_5960c29566a6.slice/crio-80362c1846ccda47de24c2fc90d304072e9f37a0b1b54fb08faae2f8d8355d8f WatchSource:0}: Error finding container 80362c1846ccda47de24c2fc90d304072e9f37a0b1b54fb08faae2f8d8355d8f: Status 404 returned error can't find the container with id 80362c1846ccda47de24c2fc90d304072e9f37a0b1b54fb08faae2f8d8355d8f
Jul 09 09:55:04.274268 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.274231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerStarted","Data":"b45d94462fa87f1ec58a7c85cc3bdf932b52d1ace7e4367b1a62e887f61c1345"}
Jul 09 09:55:04.274722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.274305    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:04.274722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.274338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerStarted","Data":"a1458349349ad9e75de238772145c80f4facc04241aee2cb96732decdc9960d4"}
Jul 09 09:55:04.274722 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.274374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerStarted","Data":"80362c1846ccda47de24c2fc90d304072e9f37a0b1b54fb08faae2f8d8355d8f"}
Jul 09 09:55:04.276201 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.276156    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerID="f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a" exitCode=2
Jul 09 09:55:04.276319 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.276201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerDied","Data":"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"}
Jul 09 09:55:04.295107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:04.295067    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podStartSLOduration=1.295051544 podStartE2EDuration="1.295051544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:04.293247569 +0000 UTC m=+1289.329057271" watchObservedRunningTime="2026-07-09 09:55:04.295051544 +0000 UTC m=+1289.330861248"
Jul 09 09:55:05.071105 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:05.071068    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:05.194510 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:05.194476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:05.281685 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:05.281652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:05.283087 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:05.283062    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:06.147367 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.147340    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:55:06.288226 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288130    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerID="67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4" exitCode=0
Jul 09 09:55:06.288226 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288205    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"
Jul 09 09:55:06.288621 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerDied","Data":"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"}
Jul 09 09:55:06.288621 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp" event={"ID":"0c5d66b0-d347-4b61-bbf0-e000397703be","Type":"ContainerDied","Data":"6b0ed7d6f6418c1cc01a7a2e7496d9efd4c0db2f9925e22306661a55aa85270d"}
Jul 09 09:55:06.288621 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288348    2576 scope.go:117] "RemoveContainer" containerID="f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"
Jul 09 09:55:06.288750 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.288674    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:06.293137 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.293115    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls\") pod \"0c5d66b0-d347-4b61-bbf0-e000397703be\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") "
Jul 09 09:55:06.293252 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.293159    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config\") pod \"0c5d66b0-d347-4b61-bbf0-e000397703be\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") "
Jul 09 09:55:06.293252 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.293224    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxtq9\" (UniqueName: \"kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9\") pod \"0c5d66b0-d347-4b61-bbf0-e000397703be\" (UID: \"0c5d66b0-d347-4b61-bbf0-e000397703be\") "
Jul 09 09:55:06.293552 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.293525    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c5d66b0-d347-4b61-bbf0-e000397703be" (UID: "0c5d66b0-d347-4b61-bbf0-e000397703be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:55:06.295286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.295257    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9" (OuterVolumeSpecName: "kube-api-access-vxtq9") pod "0c5d66b0-d347-4b61-bbf0-e000397703be" (UID: "0c5d66b0-d347-4b61-bbf0-e000397703be"). InnerVolumeSpecName "kube-api-access-vxtq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:55:06.295382 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.295301    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c5d66b0-d347-4b61-bbf0-e000397703be" (UID: "0c5d66b0-d347-4b61-bbf0-e000397703be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:55:06.296599 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.296580    2576 scope.go:117] "RemoveContainer" containerID="67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"
Jul 09 09:55:06.309921 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.309899    2576 scope.go:117] "RemoveContainer" containerID="f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"
Jul 09 09:55:06.310220 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:06.310200    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a\": container with ID starting with f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a not found: ID does not exist" containerID="f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"
Jul 09 09:55:06.310290 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.310229    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a"} err="failed to get container status \"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a\": rpc error: code = NotFound desc = could not find container \"f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a\": container with ID starting with f7c0e384398c4bbbdd98a79b3b45e0d29f26115cd2b33b618c02f3c68b62380a not found: ID does not exist"
Jul 09 09:55:06.310290 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.310247    2576 scope.go:117] "RemoveContainer" containerID="67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"
Jul 09 09:55:06.310524 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:06.310506    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4\": container with ID starting with 67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4 not found: ID does not exist" containerID="67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"
Jul 09 09:55:06.310570 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.310539    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4"} err="failed to get container status \"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4\": rpc error: code = NotFound desc = could not find container \"67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4\": container with ID starting with 67a29b1d7836ee3b72ee99d0836898896e4620a734a2f3879969805550f4e7f4 not found: ID does not exist"
Jul 09 09:55:06.394743 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.394720    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c5d66b0-d347-4b61-bbf0-e000397703be-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:06.394743 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.394742    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:06.394897 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.394751    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtq9\" (UniqueName: \"kubernetes.io/projected/0c5d66b0-d347-4b61-bbf0-e000397703be-kube-api-access-vxtq9\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:06.610948 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.610914    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:55:06.614631 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:06.614604    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b307-predictor-59c78cddf5-l7zxp"]
Jul 09 09:55:07.442214 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442163    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:07.442604 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442537    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kube-rbac-proxy"
Jul 09 09:55:07.442604 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442549    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kube-rbac-proxy"
Jul 09 09:55:07.442604 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442558    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container"
Jul 09 09:55:07.442604 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442564    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container"
Jul 09 09:55:07.442742 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442617    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kube-rbac-proxy"
Jul 09 09:55:07.442742 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.442626    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" containerName="kserve-container"
Jul 09 09:55:07.446542 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.446513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.449004 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.448981    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b04db-kube-rbac-proxy-sar-config\""
Jul 09 09:55:07.449138 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.449023    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b04db-serving-cert\""
Jul 09 09:55:07.453094 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.453067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:07.454683 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.454666    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:07.541524 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.541491    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5d66b0-d347-4b61-bbf0-e000397703be" path="/var/lib/kubelet/pods/0c5d66b0-d347-4b61-bbf0-e000397703be/volumes"
Jul 09 09:55:07.604881 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.604844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.604881 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.604880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.705426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.705343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.705426 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.705377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.705617 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:07.705508    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-b04db-serving-cert: secret "ensemble-graph-b04db-serving-cert" not found
Jul 09 09:55:07.705617 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:07.705559    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls podName:285e714e-87ec-43b3-b83c-f53fb2d7dca7 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:08.205543911 +0000 UTC m=+1293.241353594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls") pod "ensemble-graph-b04db-d79cb4dd-kpjls" (UID: "285e714e-87ec-43b3-b83c-f53fb2d7dca7") : secret "ensemble-graph-b04db-serving-cert" not found
Jul 09 09:55:07.705947 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.705928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:07.757326 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:07.757289    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:08.210540 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:08.210499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:08.212850 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:08.212829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") pod \"ensemble-graph-b04db-d79cb4dd-kpjls\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") " pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:08.357444 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:08.357419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:08.486800 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:08.486779    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:08.488276 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:55:08.488247    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285e714e_87ec_43b3_b83c_f53fb2d7dca7.slice/crio-51fb481f7da357210eb0276ccde1d19d8c6a5639ba2d3f7efc86e9212252b7fc WatchSource:0}: Error finding container 51fb481f7da357210eb0276ccde1d19d8c6a5639ba2d3f7efc86e9212252b7fc: Status 404 returned error can't find the container with id 51fb481f7da357210eb0276ccde1d19d8c6a5639ba2d3f7efc86e9212252b7fc
Jul 09 09:55:09.306229 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:09.306176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" event={"ID":"285e714e-87ec-43b3-b83c-f53fb2d7dca7","Type":"ContainerStarted","Data":"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"}
Jul 09 09:55:09.306441 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:09.306244    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" event={"ID":"285e714e-87ec-43b3-b83c-f53fb2d7dca7","Type":"ContainerStarted","Data":"51fb481f7da357210eb0276ccde1d19d8c6a5639ba2d3f7efc86e9212252b7fc"}
Jul 09 09:55:09.306441 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:09.306265    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:09.323637 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:09.323587    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podStartSLOduration=2.32357178 podStartE2EDuration="2.32357178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:09.321813978 +0000 UTC m=+1294.357623683" watchObservedRunningTime="2026-07-09 09:55:09.32357178 +0000 UTC m=+1294.359381485"
Jul 09 09:55:10.071092 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:10.071052    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:11.293530 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:11.293502    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:11.293936 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:11.293908    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:15.070808 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:15.070767    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:15.071291 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:15.070885    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:55:15.316234 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:15.316205    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:17.518748 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.518661    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:17.519230 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.518906    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" containerID="cri-o://54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5" gracePeriod=30
Jul 09 09:55:17.539534 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.539482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:17.660333 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.660295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:17.694164 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.694129    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:55:17.694460 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.694429    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container" containerID="cri-o://3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114" gracePeriod=30
Jul 09 09:55:17.694583 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.694557    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kube-rbac-proxy" containerID="cri-o://6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2" gracePeriod=30
Jul 09 09:55:17.798030 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.797950    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 09:55:17.801969 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.801947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.804272 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.804254    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42ebc-predictor-serving-cert\""
Jul 09 09:55:17.804367 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.804281    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42ebc-kube-rbac-proxy-sar-config\""
Jul 09 09:55:17.813254 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.813233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 09:55:17.892113 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.892087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.892251 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.892129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.892251 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.892152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9lg\" (UniqueName: \"kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.993534 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.993505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.993705 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.993548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.993705 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.993569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9lg\" (UniqueName: \"kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.994201 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.994144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:17.996187 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:17.996137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:18.001384 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.001362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9lg\" (UniqueName: \"kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg\") pod \"error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") " pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:18.113577 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.113502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:18.240593 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.240571    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 09:55:18.242620 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:55:18.242587    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98131af6_44e2_42e6_a79c_98e85df05f66.slice/crio-e5cd20c7da1dece78cc23222b9a0cdaf1741810ff9df184fea5071fe6dd712c4 WatchSource:0}: Error finding container e5cd20c7da1dece78cc23222b9a0cdaf1741810ff9df184fea5071fe6dd712c4: Status 404 returned error can't find the container with id e5cd20c7da1dece78cc23222b9a0cdaf1741810ff9df184fea5071fe6dd712c4
Jul 09 09:55:18.345545 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.345520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerStarted","Data":"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"}
Jul 09 09:55:18.345659 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.345584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerStarted","Data":"e5cd20c7da1dece78cc23222b9a0cdaf1741810ff9df184fea5071fe6dd712c4"}
Jul 09 09:55:18.347289 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.347263    2576 generic.go:356] "Generic (PLEG): container finished" podID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerID="6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2" exitCode=2
Jul 09 09:55:18.347374 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.347332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerDied","Data":"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"}
Jul 09 09:55:18.663389 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.663356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:18.726109 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:18.726069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:55:19.353895 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:19.353863    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerStarted","Data":"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"}
Jul 09 09:55:19.354076 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:19.353949    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:19.354076 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:19.353997    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:19.355506 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:19.355478    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:55:19.372763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:19.372709    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podStartSLOduration=2.372693101 podStartE2EDuration="2.372693101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:19.371561849 +0000 UTC m=+1304.407371566" watchObservedRunningTime="2026-07-09 09:55:19.372693101 +0000 UTC m=+1304.408502805"
Jul 09 09:55:20.071500 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:20.071458    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:20.314343 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:20.314307    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:20.359107 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:20.359024    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:55:21.060151 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.060127    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:55:21.220341 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.220319    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config\") pod \"1145536a-0c40-4de2-8755-d80cda6b21ba\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") "
Jul 09 09:55:21.220756 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.220359    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls\") pod \"1145536a-0c40-4de2-8755-d80cda6b21ba\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") "
Jul 09 09:55:21.220756 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.220384    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxshz\" (UniqueName: \"kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz\") pod \"1145536a-0c40-4de2-8755-d80cda6b21ba\" (UID: \"1145536a-0c40-4de2-8755-d80cda6b21ba\") "
Jul 09 09:55:21.220756 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.220662    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1145536a-0c40-4de2-8755-d80cda6b21ba" (UID: "1145536a-0c40-4de2-8755-d80cda6b21ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:55:21.222411 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.222390    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1145536a-0c40-4de2-8755-d80cda6b21ba" (UID: "1145536a-0c40-4de2-8755-d80cda6b21ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:55:21.222526 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.222510    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz" (OuterVolumeSpecName: "kube-api-access-wxshz") pod "1145536a-0c40-4de2-8755-d80cda6b21ba" (UID: "1145536a-0c40-4de2-8755-d80cda6b21ba"). InnerVolumeSpecName "kube-api-access-wxshz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:55:21.293957 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.293920    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:21.321571 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.321548    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1145536a-0c40-4de2-8755-d80cda6b21ba-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:21.321687 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.321573    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxshz\" (UniqueName: \"kubernetes.io/projected/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-api-access-wxshz\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:21.321687 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.321587    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1145536a-0c40-4de2-8755-d80cda6b21ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:21.365382 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.365353    2576 generic.go:356] "Generic (PLEG): container finished" podID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerID="3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114" exitCode=0
Jul 09 09:55:21.365568 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.365422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerDied","Data":"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"}
Jul 09 09:55:21.365568 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.365430    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"
Jul 09 09:55:21.365568 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.365493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" event={"ID":"1145536a-0c40-4de2-8755-d80cda6b21ba","Type":"ContainerDied","Data":"e887a2a3973a696badb58822204f52ad9b00a2fae419d93206605afc0ee800b4"}
Jul 09 09:55:21.365568 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.365518    2576 scope.go:117] "RemoveContainer" containerID="6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"
Jul 09 09:55:21.377053 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.377029    2576 scope.go:117] "RemoveContainer" containerID="3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"
Jul 09 09:55:21.384672 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.384655    2576 scope.go:117] "RemoveContainer" containerID="6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"
Jul 09 09:55:21.384919 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:21.384902    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2\": container with ID starting with 6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2 not found: ID does not exist" containerID="6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"
Jul 09 09:55:21.384965 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.384928    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2"} err="failed to get container status \"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2\": rpc error: code = NotFound desc = could not find container \"6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2\": container with ID starting with 6fc2b15a567ebfdd3eb9c5fbd61bb524b0c61b61c24487b7f3dda4974062d6b2 not found: ID does not exist"
Jul 09 09:55:21.384965 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.384946    2576 scope.go:117] "RemoveContainer" containerID="3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"
Jul 09 09:55:21.385205 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:21.385169    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114\": container with ID starting with 3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114 not found: ID does not exist" containerID="3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"
Jul 09 09:55:21.385255 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.385212    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114"} err="failed to get container status \"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114\": rpc error: code = NotFound desc = could not find container \"3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114\": container with ID starting with 3c166e10e3248e217212190ebc99f78450e76b0a558a823ac8d34a60df092114 not found: ID does not exist"
Jul 09 09:55:21.388098 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.388075    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:55:21.392388 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.392367    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc"]
Jul 09 09:55:21.541138 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:21.541060    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" path="/var/lib/kubelet/pods/1145536a-0c40-4de2-8755-d80cda6b21ba/volumes"
Jul 09 09:55:22.032432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:22.032385    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b04db-predictor-f486cdf95-s8fmc" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": context deadline exceeded"
Jul 09 09:55:25.071030 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:25.070991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:25.314344 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:25.314308    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:25.363636 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:25.363569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:55:25.363957 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:25.363935    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:55:30.070892 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:30.070855    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:30.314627 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:30.314590    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:30.314813 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:30.314716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:31.294158 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:31.294115    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:33.421916 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.421822    2576 generic.go:356] "Generic (PLEG): container finished" podID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerID="7cf6247da285afc79d0337a1e1baa919d302c8396893851936c6b34d122b2415" exitCode=0
Jul 09 09:55:33.421916 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.421895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" event={"ID":"4412a1d7-206b-4ee4-ac75-5368eeb73be8","Type":"ContainerDied","Data":"7cf6247da285afc79d0337a1e1baa919d302c8396893851936c6b34d122b2415"}
Jul 09 09:55:33.782940 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.782914    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:55:33.823725 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.823698    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle\") pod \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") "
Jul 09 09:55:33.823853 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.823798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls\") pod \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\" (UID: \"4412a1d7-206b-4ee4-ac75-5368eeb73be8\") "
Jul 09 09:55:33.824065 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.824045    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4412a1d7-206b-4ee4-ac75-5368eeb73be8" (UID: "4412a1d7-206b-4ee4-ac75-5368eeb73be8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:55:33.825700 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.825678    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4412a1d7-206b-4ee4-ac75-5368eeb73be8" (UID: "4412a1d7-206b-4ee4-ac75-5368eeb73be8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:55:33.924552 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.924520    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4412a1d7-206b-4ee4-ac75-5368eeb73be8-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:33.924552 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:33.924547    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412a1d7-206b-4ee4-ac75-5368eeb73be8-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:34.427712 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:34.427670    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"
Jul 09 09:55:34.427712 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:34.427688    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl" event={"ID":"4412a1d7-206b-4ee4-ac75-5368eeb73be8","Type":"ContainerDied","Data":"0485fc9339ed300a307ebe735cf083d0768950514a440a1a7faf0d717ab05f2e"}
Jul 09 09:55:34.428404 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:34.427753    2576 scope.go:117] "RemoveContainer" containerID="7cf6247da285afc79d0337a1e1baa919d302c8396893851936c6b34d122b2415"
Jul 09 09:55:34.453149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:34.453120    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:55:34.459399 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:34.459368    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b307-86775f9586-b64sl"]
Jul 09 09:55:35.313814 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:35.313778    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:35.363964 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:35.363925    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:55:35.541413 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:35.541383    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" path="/var/lib/kubelet/pods/4412a1d7-206b-4ee4-ac75-5368eeb73be8/volumes"
Jul 09 09:55:40.314750 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:40.314710    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:41.294464 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:41.294424    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 09:55:45.313885 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:45.313836    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:55:45.364944 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:45.364911    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:55:47.694831 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.694805    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:47.737570 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.737541    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle\") pod \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") "
Jul 09 09:55:47.737699 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.737606    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") pod \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\" (UID: \"285e714e-87ec-43b3-b83c-f53fb2d7dca7\") "
Jul 09 09:55:47.737920 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.737900    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "285e714e-87ec-43b3-b83c-f53fb2d7dca7" (UID: "285e714e-87ec-43b3-b83c-f53fb2d7dca7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:55:47.739528 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.739504    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "285e714e-87ec-43b3-b83c-f53fb2d7dca7" (UID: "285e714e-87ec-43b3-b83c-f53fb2d7dca7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:55:47.838511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.838441    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/285e714e-87ec-43b3-b83c-f53fb2d7dca7-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:47.838511 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:47.838470    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285e714e-87ec-43b3-b83c-f53fb2d7dca7-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:55:48.491382 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.491352    2576 generic.go:356] "Generic (PLEG): container finished" podID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerID="54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5" exitCode=0
Jul 09 09:55:48.491547 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.491402    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"
Jul 09 09:55:48.491547 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.491418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" event={"ID":"285e714e-87ec-43b3-b83c-f53fb2d7dca7","Type":"ContainerDied","Data":"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"}
Jul 09 09:55:48.491547 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.491457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls" event={"ID":"285e714e-87ec-43b3-b83c-f53fb2d7dca7","Type":"ContainerDied","Data":"51fb481f7da357210eb0276ccde1d19d8c6a5639ba2d3f7efc86e9212252b7fc"}
Jul 09 09:55:48.491547 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.491478    2576 scope.go:117] "RemoveContainer" containerID="54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"
Jul 09 09:55:48.499938 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.499921    2576 scope.go:117] "RemoveContainer" containerID="54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"
Jul 09 09:55:48.500205 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:55:48.500173    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5\": container with ID starting with 54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5 not found: ID does not exist" containerID="54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"
Jul 09 09:55:48.500279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.500214    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5"} err="failed to get container status \"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5\": rpc error: code = NotFound desc = could not find container \"54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5\": container with ID starting with 54b4d8695c06a161abbfe427818c6a5bffc2314bd40454945b73b8fbc8b340e5 not found: ID does not exist"
Jul 09 09:55:48.514014 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.513991    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:48.517701 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:48.517681    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b04db-d79cb4dd-kpjls"]
Jul 09 09:55:49.540781 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:49.540742    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" path="/var/lib/kubelet/pods/285e714e-87ec-43b3-b83c-f53fb2d7dca7/volumes"
Jul 09 09:55:51.295271 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:51.295242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:55:55.364545 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:55:55.364506    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 09:56:03.329077 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.328997    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329376    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329391    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329412    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329418    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329425    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329431    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329442    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kube-rbac-proxy"
Jul 09 09:56:03.329452 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329447    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kube-rbac-proxy"
Jul 09 09:56:03.329703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329507    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kube-rbac-proxy"
Jul 09 09:56:03.329703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329521    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1145536a-0c40-4de2-8755-d80cda6b21ba" containerName="kserve-container"
Jul 09 09:56:03.329703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329529    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="285e714e-87ec-43b3-b83c-f53fb2d7dca7" containerName="ensemble-graph-b04db"
Jul 09 09:56:03.329703 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.329538    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4412a1d7-206b-4ee4-ac75-5368eeb73be8" containerName="sequence-graph-8b307"
Jul 09 09:56:03.333498 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.333459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:03.333763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.333746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.336331 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.336309    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3d13-serving-cert\""
Jul 09 09:56:03.336476 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.336314    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3d13-kube-rbac-proxy-sar-config\""
Jul 09 09:56:03.341431 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.341406    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:03.431420 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.431377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:03.457085 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.457060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.457268 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.457109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.557578 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.557543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.557772 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.557629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.558208 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.558160    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.560032 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.560010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls\") pod \"sequence-graph-a3d13-549b5f46dd-xcdvl\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") " pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.645140 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.645070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:03.771229 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:03.771206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:03.773368 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:56:03.773338    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84678253_d6d6_45ec_a027_1c7160071e8e.slice/crio-86a688d6b781526df474d9becb0f1a227e3c5c043c9676dc565bf976c9e08613 WatchSource:0}: Error finding container 86a688d6b781526df474d9becb0f1a227e3c5c043c9676dc565bf976c9e08613: Status 404 returned error can't find the container with id 86a688d6b781526df474d9becb0f1a227e3c5c043c9676dc565bf976c9e08613
Jul 09 09:56:04.563357 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:04.563317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" event={"ID":"84678253-d6d6-45ec-a027-1c7160071e8e","Type":"ContainerStarted","Data":"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"}
Jul 09 09:56:04.563763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:04.563386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" event={"ID":"84678253-d6d6-45ec-a027-1c7160071e8e","Type":"ContainerStarted","Data":"86a688d6b781526df474d9becb0f1a227e3c5c043c9676dc565bf976c9e08613"}
Jul 09 09:56:04.563763 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:04.563417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:04.579369 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:04.579324    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podStartSLOduration=1.579310706 podStartE2EDuration="1.579310706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:56:04.57876898 +0000 UTC m=+1349.614578686" watchObservedRunningTime="2026-07-09 09:56:04.579310706 +0000 UTC m=+1349.615120411"
Jul 09 09:56:05.365200 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:05.365156    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 09:56:10.574279 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:10.574235    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:13.402727 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.402693    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:13.403142 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.402903    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" containerID="cri-o://9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6" gracePeriod=30
Jul 09 09:56:13.428459 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.428432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:13.485822 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.485795    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:56:13.486060 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.486017    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container" containerID="cri-o://a1458349349ad9e75de238772145c80f4facc04241aee2cb96732decdc9960d4" gracePeriod=30
Jul 09 09:56:13.486135 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.486043    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kube-rbac-proxy" containerID="cri-o://b45d94462fa87f1ec58a7c85cc3bdf932b52d1ace7e4367b1a62e887f61c1345" gracePeriod=30
Jul 09 09:56:13.511817 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.511782    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:13.583398 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.583365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:13.603766 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.603743    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerID="b45d94462fa87f1ec58a7c85cc3bdf932b52d1ace7e4367b1a62e887f61c1345" exitCode=2
Jul 09 09:56:13.603872 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.603798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerDied","Data":"b45d94462fa87f1ec58a7c85cc3bdf932b52d1ace7e4367b1a62e887f61c1345"}
Jul 09 09:56:13.682926 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.682897    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 09:56:13.686974 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.686957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.689779 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.689729    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2cd6b-predictor-serving-cert\""
Jul 09 09:56:13.690081 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.690053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:13.690081 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.690079    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2cd6b-kube-rbac-proxy-sar-config\""
Jul 09 09:56:13.696581 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.696559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 09:56:13.771982 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.771948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:13.843582 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.843551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gds4\" (UniqueName: \"kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.843736 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.843594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.843736 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.843709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.944375 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.944303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.944375 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.944355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gds4\" (UniqueName: \"kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.944534 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.944379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.944967 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.944946    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.946658 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.946637    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.953315 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.953282    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gds4\" (UniqueName: \"kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4\") pod \"error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") " pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:13.997986 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:13.997965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:14.130148 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.130114    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 09:56:14.132631 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:56:14.132602    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaad733_fd9e_4540_a67a_0e37f9a91984.slice/crio-c9d8211115b179c3bf8d690a45203f25036159569da839e7572db20d232a1560 WatchSource:0}: Error finding container c9d8211115b179c3bf8d690a45203f25036159569da839e7572db20d232a1560: Status 404 returned error can't find the container with id c9d8211115b179c3bf8d690a45203f25036159569da839e7572db20d232a1560
Jul 09 09:56:14.254159 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.254136    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:14.610121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.610038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerStarted","Data":"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"}
Jul 09 09:56:14.610121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.610095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerStarted","Data":"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"}
Jul 09 09:56:14.610121 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.610121    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:14.610555 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.610139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerStarted","Data":"c9d8211115b179c3bf8d690a45203f25036159569da839e7572db20d232a1560"}
Jul 09 09:56:14.628467 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:14.628411    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podStartSLOduration=1.6283955300000001 podStartE2EDuration="1.62839553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:56:14.626844641 +0000 UTC m=+1359.662654345" watchObservedRunningTime="2026-07-09 09:56:14.62839553 +0000 UTC m=+1359.664205236"
Jul 09 09:56:15.572409 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:15.572371    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:15.614584 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:15.614557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:15.615732 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:15.615706    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:16.289199 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.289154    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 09 09:56:16.621640 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.621613    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerID="a1458349349ad9e75de238772145c80f4facc04241aee2cb96732decdc9960d4" exitCode=0
Jul 09 09:56:16.621992 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.621689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerDied","Data":"a1458349349ad9e75de238772145c80f4facc04241aee2cb96732decdc9960d4"}
Jul 09 09:56:16.622149 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.622124    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:16.643226 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.643207    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:56:16.770416 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.770377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls\") pod \"dd8634d0-e555-4d48-bccd-5960c29566a6\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") "
Jul 09 09:56:16.770583 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.770458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnk5z\" (UniqueName: \"kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z\") pod \"dd8634d0-e555-4d48-bccd-5960c29566a6\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") "
Jul 09 09:56:16.770583 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.770499    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config\") pod \"dd8634d0-e555-4d48-bccd-5960c29566a6\" (UID: \"dd8634d0-e555-4d48-bccd-5960c29566a6\") "
Jul 09 09:56:16.770891 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.770855    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd8634d0-e555-4d48-bccd-5960c29566a6" (UID: "dd8634d0-e555-4d48-bccd-5960c29566a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:56:16.772542 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.772521    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd8634d0-e555-4d48-bccd-5960c29566a6" (UID: "dd8634d0-e555-4d48-bccd-5960c29566a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:56:16.772542 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.772536    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z" (OuterVolumeSpecName: "kube-api-access-bnk5z") pod "dd8634d0-e555-4d48-bccd-5960c29566a6" (UID: "dd8634d0-e555-4d48-bccd-5960c29566a6"). InnerVolumeSpecName "kube-api-access-bnk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:56:16.871559 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.871477    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8634d0-e555-4d48-bccd-5960c29566a6-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:56:16.871559 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.871506    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnk5z\" (UniqueName: \"kubernetes.io/projected/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-api-access-bnk5z\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:56:16.871559 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:16.871515    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8634d0-e555-4d48-bccd-5960c29566a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:56:17.627692 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.627655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl" event={"ID":"dd8634d0-e555-4d48-bccd-5960c29566a6","Type":"ContainerDied","Data":"80362c1846ccda47de24c2fc90d304072e9f37a0b1b54fb08faae2f8d8355d8f"}
Jul 09 09:56:17.627692 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.627686    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"
Jul 09 09:56:17.628205 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.627727    2576 scope.go:117] "RemoveContainer" containerID="b45d94462fa87f1ec58a7c85cc3bdf932b52d1ace7e4367b1a62e887f61c1345"
Jul 09 09:56:17.636023 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.636003    2576 scope.go:117] "RemoveContainer" containerID="a1458349349ad9e75de238772145c80f4facc04241aee2cb96732decdc9960d4"
Jul 09 09:56:17.646246 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.646223    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:56:17.654352 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.654331    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d13-predictor-58658c6cf9-pr7kl"]
Jul 09 09:56:17.775759 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.775729    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 09:56:17.776106 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776093    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container"
Jul 09 09:56:17.776146 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776107    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container"
Jul 09 09:56:17.776146 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776134    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kube-rbac-proxy"
Jul 09 09:56:17.776146 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776139    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kube-rbac-proxy"
Jul 09 09:56:17.776284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776255    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kube-rbac-proxy"
Jul 09 09:56:17.776284 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.776267    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" containerName="kserve-container"
Jul 09 09:56:17.777975 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.777949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:17.780818 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.780801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.783506 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.783487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-42ebc-kube-rbac-proxy-sar-config\""
Jul 09 09:56:17.783607 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.783487    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-42ebc-serving-cert\""
Jul 09 09:56:17.796866 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.795057    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 09:56:17.872286 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.872170    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:56:17.878347 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.878315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.878494 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.878362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.979675 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.979642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.979815 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.979679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.980310 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.980290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:17.982154 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:17.982122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls\") pod \"ensemble-graph-42ebc-7bff6d66c7-qkhvz\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") " pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:18.090641 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.090606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:18.217928 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.217897    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 09:56:18.220561 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:56:18.220537    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77961d8f_cadf_4169_b98c_5ebda502652f.slice/crio-71ed942f2a3ac25767f64718182bc64991405e2a73f2c9c88d9f524c5944a628 WatchSource:0}: Error finding container 71ed942f2a3ac25767f64718182bc64991405e2a73f2c9c88d9f524c5944a628: Status 404 returned error can't find the container with id 71ed942f2a3ac25767f64718182bc64991405e2a73f2c9c88d9f524c5944a628
Jul 09 09:56:18.634311 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.634276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" event={"ID":"77961d8f-cadf-4169-b98c-5ebda502652f","Type":"ContainerStarted","Data":"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"}
Jul 09 09:56:18.634797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.634432    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:18.634797 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.634479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" event={"ID":"77961d8f-cadf-4169-b98c-5ebda502652f","Type":"ContainerStarted","Data":"71ed942f2a3ac25767f64718182bc64991405e2a73f2c9c88d9f524c5944a628"}
Jul 09 09:56:18.669089 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:18.669043    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podStartSLOduration=1.6690237460000001 podStartE2EDuration="1.669023746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:56:18.667629001 +0000 UTC m=+1363.703438705" watchObservedRunningTime="2026-07-09 09:56:18.669023746 +0000 UTC m=+1363.704833451"
Jul 09 09:56:19.542282 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:19.542249    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8634d0-e555-4d48-bccd-5960c29566a6" path="/var/lib/kubelet/pods/dd8634d0-e555-4d48-bccd-5960c29566a6/volumes"
Jul 09 09:56:20.572480 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:20.572445    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:21.627002 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:21.626974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:56:21.627497 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:21.627469    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:24.644730 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:24.644686    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 09:56:25.572109 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:25.572070    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:25.572312 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:25.572164    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:30.571625 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:30.571585    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:31.627990 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:31.627952    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:35.571752 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:35.571719    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:40.571640 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:40.571601    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:56:41.628019 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:41.627969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:43.586147 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.586126    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:43.691289 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.691266    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle\") pod \"84678253-d6d6-45ec-a027-1c7160071e8e\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") "
Jul 09 09:56:43.691415 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.691320    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls\") pod \"84678253-d6d6-45ec-a027-1c7160071e8e\" (UID: \"84678253-d6d6-45ec-a027-1c7160071e8e\") "
Jul 09 09:56:43.691663 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.691637    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "84678253-d6d6-45ec-a027-1c7160071e8e" (UID: "84678253-d6d6-45ec-a027-1c7160071e8e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:56:43.693296 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.693275    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84678253-d6d6-45ec-a027-1c7160071e8e" (UID: "84678253-d6d6-45ec-a027-1c7160071e8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:56:43.750130 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.750094    2576 generic.go:356] "Generic (PLEG): container finished" podID="84678253-d6d6-45ec-a027-1c7160071e8e" containerID="9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6" exitCode=0
Jul 09 09:56:43.750267 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.750141    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"
Jul 09 09:56:43.750267 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.750147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" event={"ID":"84678253-d6d6-45ec-a027-1c7160071e8e","Type":"ContainerDied","Data":"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"}
Jul 09 09:56:43.750267 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.750221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl" event={"ID":"84678253-d6d6-45ec-a027-1c7160071e8e","Type":"ContainerDied","Data":"86a688d6b781526df474d9becb0f1a227e3c5c043c9676dc565bf976c9e08613"}
Jul 09 09:56:43.750267 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.750245    2576 scope.go:117] "RemoveContainer" containerID="9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"
Jul 09 09:56:43.758971 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.758957    2576 scope.go:117] "RemoveContainer" containerID="9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"
Jul 09 09:56:43.759283 ip-10-0-141-211 kubenswrapper[2576]: E0709 09:56:43.759256    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6\": container with ID starting with 9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6 not found: ID does not exist" containerID="9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"
Jul 09 09:56:43.759375 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.759293    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6"} err="failed to get container status \"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6\": rpc error: code = NotFound desc = could not find container \"9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6\": container with ID starting with 9395c50c4fb394b460444c737a9c2a28b32f26214fa0f178263d2dfc8eabd2b6 not found: ID does not exist"
Jul 09 09:56:43.773257 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.773235    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:43.777696 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.777677    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3d13-549b5f46dd-xcdvl"]
Jul 09 09:56:43.792493 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.792475    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84678253-d6d6-45ec-a027-1c7160071e8e-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:56:43.792558 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:43.792494    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84678253-d6d6-45ec-a027-1c7160071e8e-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 09:56:45.547697 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:45.547665    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" path="/var/lib/kubelet/pods/84678253-d6d6-45ec-a027-1c7160071e8e/volumes"
Jul 09 09:56:51.628349 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:51.628312    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 09:56:59.415755 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:56:59.415708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:57:01.627904 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:01.627874    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 09:57:13.660109 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.660077    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 09:57:13.660660 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.660439    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13"
Jul 09 09:57:13.660660 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.660451    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13"
Jul 09 09:57:13.660660 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.660524    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84678253-d6d6-45ec-a027-1c7160071e8e" containerName="sequence-graph-a3d13"
Jul 09 09:57:13.662002 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.661975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:57:13.663414 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.663398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.665876 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.665857    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2cd6b-serving-cert\""
Jul 09 09:57:13.666432 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.666407    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2cd6b-kube-rbac-proxy-sar-config\""
Jul 09 09:57:13.672275 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.672254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 09:57:13.751055 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.751025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.751200 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.751130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.789986 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.789954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 09:57:13.852235 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.852198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.852377 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.852241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.852867 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.852849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.854487 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.854468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls\") pod \"sequence-graph-2cd6b-748d5dcdf4-8jbw4\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") " pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:13.975141 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:13.975111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:14.109129 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:14.109100    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 09:57:14.111314 ip-10-0-141-211 kubenswrapper[2576]: W0709 09:57:14.111287    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc57c84_5d6a_4fdc_afe6_c359baf509ec.slice/crio-cf8ba804ca9060c68c4bd34b6c660d3a8e7221099e833d541f8b7e02fbc0b7d4 WatchSource:0}: Error finding container cf8ba804ca9060c68c4bd34b6c660d3a8e7221099e833d541f8b7e02fbc0b7d4: Status 404 returned error can't find the container with id cf8ba804ca9060c68c4bd34b6c660d3a8e7221099e833d541f8b7e02fbc0b7d4
Jul 09 09:57:14.887345 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:14.887306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" event={"ID":"abc57c84-5d6a-4fdc-afe6-c359baf509ec","Type":"ContainerStarted","Data":"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"}
Jul 09 09:57:14.887836 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:14.887367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" event={"ID":"abc57c84-5d6a-4fdc-afe6-c359baf509ec","Type":"ContainerStarted","Data":"cf8ba804ca9060c68c4bd34b6c660d3a8e7221099e833d541f8b7e02fbc0b7d4"}
Jul 09 09:57:14.887836 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:14.887444    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:14.903996 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:14.903952    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podStartSLOduration=1.903937899 podStartE2EDuration="1.903937899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:57:14.902966685 +0000 UTC m=+1419.938776389" watchObservedRunningTime="2026-07-09 09:57:14.903937899 +0000 UTC m=+1419.939747603"
Jul 09 09:57:20.897439 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:20.897410    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 09:57:42.203600 ip-10-0-141-211 kubenswrapper[2576]: I0709 09:57:42.203566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:03:01.138587 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:03:01.138551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:00.557335 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:00.557291    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.381859 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.381821    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 10:04:32.382501 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.382032    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" containerID="cri-o://d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4" gracePeriod=30
Jul 09 10:04:32.407287 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.407258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.549390 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.549351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.617909 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.617871    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 10:04:32.618149 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.618111    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" containerID="cri-o://bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e" gracePeriod=30
Jul 09 10:04:32.618387 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.618151    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kube-rbac-proxy" containerID="cri-o://5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb" gracePeriod=30
Jul 09 10:04:32.621602 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.621580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.657402 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.657364    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:04:32.661023 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.661002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.663314 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.663294    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d678-predictor-serving-cert\""
Jul 09 10:04:32.663447 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.663423    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d678-kube-rbac-proxy-sar-config\""
Jul 09 10:04:32.670374 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.670349    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:04:32.699927 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.699904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.754237 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.754211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.754326 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.754280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddbl\" (UniqueName: \"kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.754367 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.754333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.814497 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.814469    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:32.855157 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.855120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddbl\" (UniqueName: \"kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.855314 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.855195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.855314 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.855236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.855417 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:32.855354    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9d678-predictor-serving-cert: secret "error-404-isvc-9d678-predictor-serving-cert" not found
Jul 09 10:04:32.855466 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:32.855445    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls podName:15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed nodeName:}" failed. No retries permitted until 2026-07-09 10:04:33.355414393 +0000 UTC m=+1858.391224084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls") pod "error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" (UID: "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed") : secret "error-404-isvc-9d678-predictor-serving-cert" not found
Jul 09 10:04:32.855900 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.855883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:32.859417 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.859393    2576 generic.go:356] "Generic (PLEG): container finished" podID="98131af6-44e2-42e6-a79c-98e85df05f66" containerID="5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb" exitCode=2
Jul 09 10:04:32.859503 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.859449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerDied","Data":"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"}
Jul 09 10:04:32.863914 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:32.863889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddbl\" (UniqueName: \"kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:33.359253 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:33.359214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:33.359439 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:33.359358    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9d678-predictor-serving-cert: secret "error-404-isvc-9d678-predictor-serving-cert" not found
Jul 09 10:04:33.359439 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:33.359432    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls podName:15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed nodeName:}" failed. No retries permitted until 2026-07-09 10:04:34.359416849 +0000 UTC m=+1859.395226533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls") pod "error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" (UID: "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed") : secret "error-404-isvc-9d678-predictor-serving-cert" not found
Jul 09 10:04:33.486198 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:33.486143    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:04:34.368478 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.368437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:34.370857 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.370837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"error-404-isvc-9d678-predictor-6b97f5f755-bbpfq\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") " pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:34.472063 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.472039    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:34.599919 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.599896    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:04:34.602269 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:04:34.602233    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e4fa1a_11e6_4f3a_8571_12fe9ea6f9ed.slice/crio-595ce4d61c5558341de1be1f6b0a7e6f404ca488b59f0dd59eaf97a99ad3d9ca WatchSource:0}: Error finding container 595ce4d61c5558341de1be1f6b0a7e6f404ca488b59f0dd59eaf97a99ad3d9ca: Status 404 returned error can't find the container with id 595ce4d61c5558341de1be1f6b0a7e6f404ca488b59f0dd59eaf97a99ad3d9ca
Jul 09 10:04:34.642515 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.642486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:04:34.870125 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.870091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerStarted","Data":"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"}
Jul 09 10:04:34.870334 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.870151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerStarted","Data":"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"}
Jul 09 10:04:34.870334 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.870175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerStarted","Data":"595ce4d61c5558341de1be1f6b0a7e6f404ca488b59f0dd59eaf97a99ad3d9ca"}
Jul 09 10:04:34.870334 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.870233    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:34.890357 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:34.890303    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podStartSLOduration=2.890285773 podStartE2EDuration="2.890285773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:04:34.888859306 +0000 UTC m=+1859.924669027" watchObservedRunningTime="2026-07-09 10:04:34.890285773 +0000 UTC m=+1859.926095555"
Jul 09 10:04:35.359529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.359489    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 09 10:04:35.363962 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.363933    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 10:04:35.761802 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.761776    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 10:04:35.875225 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875168    2576 generic.go:356] "Generic (PLEG): container finished" podID="98131af6-44e2-42e6-a79c-98e85df05f66" containerID="bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e" exitCode=0
Jul 09 10:04:35.875410 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875238    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"
Jul 09 10:04:35.875410 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerDied","Data":"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"}
Jul 09 10:04:35.875410 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w" event={"ID":"98131af6-44e2-42e6-a79c-98e85df05f66","Type":"ContainerDied","Data":"e5cd20c7da1dece78cc23222b9a0cdaf1741810ff9df184fea5071fe6dd712c4"}
Jul 09 10:04:35.875410 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875317    2576 scope.go:117] "RemoveContainer" containerID="5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"
Jul 09 10:04:35.875640 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.875621    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:35.877000 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.876976    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:04:35.880369 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.880314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls\") pod \"98131af6-44e2-42e6-a79c-98e85df05f66\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") "
Jul 09 10:04:35.880369 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.880350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config\") pod \"98131af6-44e2-42e6-a79c-98e85df05f66\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") "
Jul 09 10:04:35.880523 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.880384    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp9lg\" (UniqueName: \"kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg\") pod \"98131af6-44e2-42e6-a79c-98e85df05f66\" (UID: \"98131af6-44e2-42e6-a79c-98e85df05f66\") "
Jul 09 10:04:35.880681 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.880655    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98131af6-44e2-42e6-a79c-98e85df05f66" (UID: "98131af6-44e2-42e6-a79c-98e85df05f66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:04:35.882363 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.882335    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98131af6-44e2-42e6-a79c-98e85df05f66" (UID: "98131af6-44e2-42e6-a79c-98e85df05f66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:04:35.882463 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.882448    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg" (OuterVolumeSpecName: "kube-api-access-gp9lg") pod "98131af6-44e2-42e6-a79c-98e85df05f66" (UID: "98131af6-44e2-42e6-a79c-98e85df05f66"). InnerVolumeSpecName "kube-api-access-gp9lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:04:35.883966 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.883947    2576 scope.go:117] "RemoveContainer" containerID="bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"
Jul 09 10:04:35.893236 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.893221    2576 scope.go:117] "RemoveContainer" containerID="5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"
Jul 09 10:04:35.893502 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:35.893474    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb\": container with ID starting with 5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb not found: ID does not exist" containerID="5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"
Jul 09 10:04:35.893554 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.893500    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb"} err="failed to get container status \"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb\": rpc error: code = NotFound desc = could not find container \"5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb\": container with ID starting with 5803b022d43d8ef75e474b00a2b72d8cfce5dbdb22e573bf0e5998473aa3e2bb not found: ID does not exist"
Jul 09 10:04:35.893554 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.893522    2576 scope.go:117] "RemoveContainer" containerID="bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"
Jul 09 10:04:35.893779 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:04:35.893758    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e\": container with ID starting with bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e not found: ID does not exist" containerID="bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"
Jul 09 10:04:35.893826 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.893785    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e"} err="failed to get container status \"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e\": rpc error: code = NotFound desc = could not find container \"bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e\": container with ID starting with bd91af5fc721b836a123b764e421522513e11ae1f4393bdd6f6458497456f00e not found: ID does not exist"
Jul 09 10:04:35.981053 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.981024    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp9lg\" (UniqueName: \"kubernetes.io/projected/98131af6-44e2-42e6-a79c-98e85df05f66-kube-api-access-gp9lg\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:04:35.981053 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.981049    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98131af6-44e2-42e6-a79c-98e85df05f66-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:04:35.981053 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:35.981058    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98131af6-44e2-42e6-a79c-98e85df05f66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:04:36.196681 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:36.196652    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 10:04:36.202985 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:36.202962    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42ebc-predictor-5b56bdc44c-hwc6w"]
Jul 09 10:04:36.880981 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:36.880937    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:04:37.541386 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:37.541347    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" path="/var/lib/kubelet/pods/98131af6-44e2-42e6-a79c-98e85df05f66/volumes"
Jul 09 10:04:39.642482 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:39.642440    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:04:41.885717 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:41.885688    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:04:41.886111 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:41.886086    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:04:44.642520 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:44.642479    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:04:44.642916 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:44.642589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 10:04:49.642897 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:49.642860    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:04:51.887028 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:51.886988    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:04:54.642334 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:54.642295    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:04:59.642974 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:04:59.642887    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:01.886754 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:01.886711    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:05:02.535946 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.535924    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 10:05:02.599026 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.598996    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle\") pod \"77961d8f-cadf-4169-b98c-5ebda502652f\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") "
Jul 09 10:05:02.599198 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.599043    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls\") pod \"77961d8f-cadf-4169-b98c-5ebda502652f\" (UID: \"77961d8f-cadf-4169-b98c-5ebda502652f\") "
Jul 09 10:05:02.599350 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.599326    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "77961d8f-cadf-4169-b98c-5ebda502652f" (UID: "77961d8f-cadf-4169-b98c-5ebda502652f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:05:02.601037 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.601012    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77961d8f-cadf-4169-b98c-5ebda502652f" (UID: "77961d8f-cadf-4169-b98c-5ebda502652f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:05:02.699648 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.699613    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77961d8f-cadf-4169-b98c-5ebda502652f-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:02.699648 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.699648    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77961d8f-cadf-4169-b98c-5ebda502652f-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:02.995751 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.995664    2576 generic.go:356] "Generic (PLEG): container finished" podID="77961d8f-cadf-4169-b98c-5ebda502652f" containerID="d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4" exitCode=0
Jul 09 10:05:02.995751 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.995705    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"
Jul 09 10:05:02.996175 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.995753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" event={"ID":"77961d8f-cadf-4169-b98c-5ebda502652f","Type":"ContainerDied","Data":"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"}
Jul 09 10:05:02.996175 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.995804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz" event={"ID":"77961d8f-cadf-4169-b98c-5ebda502652f","Type":"ContainerDied","Data":"71ed942f2a3ac25767f64718182bc64991405e2a73f2c9c88d9f524c5944a628"}
Jul 09 10:05:02.996175 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:02.995829    2576 scope.go:117] "RemoveContainer" containerID="d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"
Jul 09 10:05:03.004324 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:03.004308    2576 scope.go:117] "RemoveContainer" containerID="d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"
Jul 09 10:05:03.004568 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:03.004549    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4\": container with ID starting with d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4 not found: ID does not exist" containerID="d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"
Jul 09 10:05:03.004610 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:03.004575    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4"} err="failed to get container status \"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4\": rpc error: code = NotFound desc = could not find container \"d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4\": container with ID starting with d4a587e17dfb90da48c8ab3ec80cb2f39f333b61be435bf4ce0e63e9777593d4 not found: ID does not exist"
Jul 09 10:05:03.016757 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:03.016736    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 10:05:03.022297 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:03.022276    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-42ebc-7bff6d66c7-qkhvz"]
Jul 09 10:05:03.540920 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:03.540888    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" path="/var/lib/kubelet/pods/77961d8f-cadf-4169-b98c-5ebda502652f/volumes"
Jul 09 10:05:11.886826 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:11.886783    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 10:05:21.886765 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:21.886731    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:05:28.373892 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.373859    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 10:05:28.374321 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.374068    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" containerID="cri-o://0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859" gracePeriod=30
Jul 09 10:05:28.409065 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.409035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:28.459572 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.459540    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 10:05:28.459864 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.459812    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container" containerID="cri-o://503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f" gracePeriod=30
Jul 09 10:05:28.460128 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.460054    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kube-rbac-proxy" containerID="cri-o://a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84" gracePeriod=30
Jul 09 10:05:28.682973 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.682950    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:05:28.683355 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683341    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc"
Jul 09 10:05:28.683405 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683359    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc"
Jul 09 10:05:28.683405 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683381    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container"
Jul 09 10:05:28.683405 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683389    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container"
Jul 09 10:05:28.683405 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683402    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kube-rbac-proxy"
Jul 09 10:05:28.683530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683408    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kube-rbac-proxy"
Jul 09 10:05:28.683530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683475    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kube-rbac-proxy"
Jul 09 10:05:28.683530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683486    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="98131af6-44e2-42e6-a79c-98e85df05f66" containerName="kserve-container"
Jul 09 10:05:28.683530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.683493    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77961d8f-cadf-4169-b98c-5ebda502652f" containerName="ensemble-graph-42ebc"
Jul 09 10:05:28.686543 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.686523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.689739 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.689719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52f9b-predictor-serving-cert\""
Jul 09 10:05:28.689845 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.689761    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52f9b-kube-rbac-proxy-sar-config\""
Jul 09 10:05:28.695900 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.695872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:05:28.814407 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.814382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5d7\" (UniqueName: \"kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.814539 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.814416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.814539 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.814454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.824968 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.824943    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:28.915938 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.915894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.916144 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.915976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5d7\" (UniqueName: \"kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.916144 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.916007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.916582 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.916556    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.918625 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.918597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.925322 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.925283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:28.925544 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.925522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5d7\" (UniqueName: \"kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7\") pod \"error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") " pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:28.996947 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:28.996869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:29.114924 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:29.114890    2576 generic.go:356] "Generic (PLEG): container finished" podID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerID="a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84" exitCode=2
Jul 09 10:05:29.115082 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:29.114958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerDied","Data":"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"}
Jul 09 10:05:29.135935 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:29.135910    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:05:29.137780 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:05:29.137749    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a70fe9_fdf0_43ef_a105_083311069702.slice/crio-5424fedcc9c49950826a84dfa5d373b81810a8982f93840c386f42da18d098b3 WatchSource:0}: Error finding container 5424fedcc9c49950826a84dfa5d373b81810a8982f93840c386f42da18d098b3: Status 404 returned error can't find the container with id 5424fedcc9c49950826a84dfa5d373b81810a8982f93840c386f42da18d098b3
Jul 09 10:05:30.121245 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.121210    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerStarted","Data":"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"}
Jul 09 10:05:30.121619 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.121269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerStarted","Data":"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"}
Jul 09 10:05:30.121619 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.121291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerStarted","Data":"5424fedcc9c49950826a84dfa5d373b81810a8982f93840c386f42da18d098b3"}
Jul 09 10:05:30.121619 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.121365    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:30.140500 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.140455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podStartSLOduration=2.140441116 podStartE2EDuration="2.140441116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:05:30.138987842 +0000 UTC m=+1915.174797561" watchObservedRunningTime="2026-07-09 10:05:30.140441116 +0000 UTC m=+1915.176250820"
Jul 09 10:05:30.895101 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:30.895066    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:31.126384 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.126356    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:31.127784 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.127747    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:05:31.617710 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.617685    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 10:05:31.740760 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.740687    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config\") pod \"bbaad733-fd9e-4540-a67a-0e37f9a91984\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") "
Jul 09 10:05:31.740760 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.740731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gds4\" (UniqueName: \"kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4\") pod \"bbaad733-fd9e-4540-a67a-0e37f9a91984\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") "
Jul 09 10:05:31.740935 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.740794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls\") pod \"bbaad733-fd9e-4540-a67a-0e37f9a91984\" (UID: \"bbaad733-fd9e-4540-a67a-0e37f9a91984\") "
Jul 09 10:05:31.741157 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.741120    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbaad733-fd9e-4540-a67a-0e37f9a91984" (UID: "bbaad733-fd9e-4540-a67a-0e37f9a91984"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:05:31.742760 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.742733    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4" (OuterVolumeSpecName: "kube-api-access-7gds4") pod "bbaad733-fd9e-4540-a67a-0e37f9a91984" (UID: "bbaad733-fd9e-4540-a67a-0e37f9a91984"). InnerVolumeSpecName "kube-api-access-7gds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:05:31.742895 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.742874    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbaad733-fd9e-4540-a67a-0e37f9a91984" (UID: "bbaad733-fd9e-4540-a67a-0e37f9a91984"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:05:31.842299 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.842266    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gds4\" (UniqueName: \"kubernetes.io/projected/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-api-access-7gds4\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:31.842299 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.842291    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbaad733-fd9e-4540-a67a-0e37f9a91984-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:31.842299 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:31.842301    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbaad733-fd9e-4540-a67a-0e37f9a91984-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:32.133213 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133109    2576 generic.go:356] "Generic (PLEG): container finished" podID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerID="503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f" exitCode=0
Jul 09 10:05:32.133213 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133157    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"
Jul 09 10:05:32.133718 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerDied","Data":"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"}
Jul 09 10:05:32.133718 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85" event={"ID":"bbaad733-fd9e-4540-a67a-0e37f9a91984","Type":"ContainerDied","Data":"c9d8211115b179c3bf8d690a45203f25036159569da839e7572db20d232a1560"}
Jul 09 10:05:32.133718 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133303    2576 scope.go:117] "RemoveContainer" containerID="a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"
Jul 09 10:05:32.133938 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.133906    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:05:32.143213 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.143193    2576 scope.go:117] "RemoveContainer" containerID="503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"
Jul 09 10:05:32.150546 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.150530    2576 scope.go:117] "RemoveContainer" containerID="a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"
Jul 09 10:05:32.150837 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:32.150819    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84\": container with ID starting with a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84 not found: ID does not exist" containerID="a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"
Jul 09 10:05:32.150909 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.150847    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84"} err="failed to get container status \"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84\": rpc error: code = NotFound desc = could not find container \"a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84\": container with ID starting with a6b8eb9e6311df1298a68ba01054aab247ae60f7daee3d30125d0fe8b4b20a84 not found: ID does not exist"
Jul 09 10:05:32.150909 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.150863    2576 scope.go:117] "RemoveContainer" containerID="503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"
Jul 09 10:05:32.151095 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:32.151076    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f\": container with ID starting with 503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f not found: ID does not exist" containerID="503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"
Jul 09 10:05:32.151140 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.151102    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f"} err="failed to get container status \"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f\": rpc error: code = NotFound desc = could not find container \"503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f\": container with ID starting with 503dc36b4fb785c53327b44ea69524bbca7645056ed6c70a8bcd051ad72d4a8f not found: ID does not exist"
Jul 09 10:05:32.155812 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.155790    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 10:05:32.162190 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.162159    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2cd6b-predictor-7d4869ff97-jqr85"]
Jul 09 10:05:32.615696 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.615665    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:05:32.616027 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616016    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container"
Jul 09 10:05:32.616072 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616028    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container"
Jul 09 10:05:32.616072 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616041    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kube-rbac-proxy"
Jul 09 10:05:32.616072 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616046    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kube-rbac-proxy"
Jul 09 10:05:32.616164 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616117    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kube-rbac-proxy"
Jul 09 10:05:32.616164 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.616125    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" containerName="kserve-container"
Jul 09 10:05:32.620469 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.620449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.622792 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.622771    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9d678-kube-rbac-proxy-sar-config\""
Jul 09 10:05:32.622912 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.622883    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9d678-serving-cert\""
Jul 09 10:05:32.628424 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.628397    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:05:32.630853 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.630827    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:32.751386 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.751356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.751550 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.751427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.773217 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.773166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:32.852115 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.852084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.852247 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.852139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.852745 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.852725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.854582 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.854559    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls\") pod \"splitter-graph-9d678-56c66cc487-c4m49\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") " pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:32.933214 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:32.933175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:33.058755 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.058660    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:05:33.061367 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:05:33.061341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5185756b_f66b_43b8_abec_4422ec5c999e.slice/crio-dee8dc9f10d3c0e2aa99fb7e461e74f0451ef4638236bd55aaa5c29cb65201fb WatchSource:0}: Error finding container dee8dc9f10d3c0e2aa99fb7e461e74f0451ef4638236bd55aaa5c29cb65201fb: Status 404 returned error can't find the container with id dee8dc9f10d3c0e2aa99fb7e461e74f0451ef4638236bd55aaa5c29cb65201fb
Jul 09 10:05:33.139586 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.139554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" event={"ID":"5185756b-f66b-43b8-abec-4422ec5c999e","Type":"ContainerStarted","Data":"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"}
Jul 09 10:05:33.139993 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.139611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" event={"ID":"5185756b-f66b-43b8-abec-4422ec5c999e","Type":"ContainerStarted","Data":"dee8dc9f10d3c0e2aa99fb7e461e74f0451ef4638236bd55aaa5c29cb65201fb"}
Jul 09 10:05:33.139993 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.139688    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:33.158485 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.158445    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podStartSLOduration=1.158430703 podStartE2EDuration="1.158430703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:05:33.15722711 +0000 UTC m=+1918.193036815" watchObservedRunningTime="2026-07-09 10:05:33.158430703 +0000 UTC m=+1918.194240407"
Jul 09 10:05:33.542147 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:33.542113    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbaad733-fd9e-4540-a67a-0e37f9a91984" path="/var/lib/kubelet/pods/bbaad733-fd9e-4540-a67a-0e37f9a91984/volumes"
Jul 09 10:05:35.895251 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:35.895143    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:37.138154 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:37.138123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:05:37.138608 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:37.138584    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:05:39.151678 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:39.151647    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:40.894551 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:40.894511    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:40.894920 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:40.894630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 10:05:42.702453 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.702420    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:05:42.703167 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.703110    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" containerID="cri-o://b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e" gracePeriod=30
Jul 09 10:05:42.718114 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.718085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:42.855251 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.855196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:42.955881 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.955780    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:05:42.956073 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.956046    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container" containerID="cri-o://b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3" gracePeriod=30
Jul 09 10:05:42.956154 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.956128    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kube-rbac-proxy" containerID="cri-o://f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939" gracePeriod=30
Jul 09 10:05:42.957753 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.957717    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:42.987066 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.987036    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:05:42.990578 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.990559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:42.992853 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.992836    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a7fc2-predictor-serving-cert\""
Jul 09 10:05:42.992955 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:42.992896    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a7fc2-kube-rbac-proxy-sar-config\""
Jul 09 10:05:43.004461 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.004440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:05:43.027334 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.027307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:43.136740 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.136702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvq4\" (UniqueName: \"kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.136904 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.136846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.136950 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.136900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.187585 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.187551    2576 generic.go:356] "Generic (PLEG): container finished" podID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerID="f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939" exitCode=2
Jul 09 10:05:43.187710 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.187624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerDied","Data":"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"}
Jul 09 10:05:43.237926 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.237846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvq4\" (UniqueName: \"kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.238070 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.237957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.238070 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.237997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.238210 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:43.238111    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-serving-cert: secret "error-404-isvc-a7fc2-predictor-serving-cert" not found
Jul 09 10:05:43.238210 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:43.238175    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls podName:3a6481a6-4e72-4e32-adb4-d4a9932d530c nodeName:}" failed. No retries permitted until 2026-07-09 10:05:43.738157155 +0000 UTC m=+1928.773966852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls") pod "error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" (UID: "3a6481a6-4e72-4e32-adb4-d4a9932d530c") : secret "error-404-isvc-a7fc2-predictor-serving-cert" not found
Jul 09 10:05:43.238759 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.238742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.247243 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.247218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvq4\" (UniqueName: \"kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.277091 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.277066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:43.588943 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.588865    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:05:43.742492 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.742461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.744845 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.744825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") pod \"error-404-isvc-a7fc2-predictor-78f845d769-bmbbs\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") " pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:43.900534 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:43.900455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:44.035329 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.035225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:05:44.037514 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:05:44.037484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6481a6_4e72_4e32_adb4_d4a9932d530c.slice/crio-cec37fda1f3898b138e6414f38626d8f19f34355c995e79df02aa811d4347cf2 WatchSource:0}: Error finding container cec37fda1f3898b138e6414f38626d8f19f34355c995e79df02aa811d4347cf2: Status 404 returned error can't find the container with id cec37fda1f3898b138e6414f38626d8f19f34355c995e79df02aa811d4347cf2
Jul 09 10:05:44.150174 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.150140    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:44.193564 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.193532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerStarted","Data":"3d373704f9b72a41337a7626140ec3576f588da8ee850f818e2460008a0d09e6"}
Jul 09 10:05:44.193683 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.193592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerStarted","Data":"812adecf8bb786e80f154b60c44216b1ca9eb8ef31536b2ac0ced2391fdf1745"}
Jul 09 10:05:44.193683 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.193617    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerStarted","Data":"cec37fda1f3898b138e6414f38626d8f19f34355c995e79df02aa811d4347cf2"}
Jul 09 10:05:44.193683 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.193664    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:44.214992 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:44.214950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podStartSLOduration=2.214938314 podStartE2EDuration="2.214938314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:05:44.213143935 +0000 UTC m=+1929.248953650" watchObservedRunningTime="2026-07-09 10:05:44.214938314 +0000 UTC m=+1929.250748019"
Jul 09 10:05:45.197890 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.197858    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:45.199141 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.199114    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:05:45.894893 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.894862    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:45.921419 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.921397    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:05:45.963910 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.963878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") pod \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") "
Jul 09 10:05:45.964057 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.963930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config\") pod \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") "
Jul 09 10:05:45.964123 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.964050    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jddbl\" (UniqueName: \"kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl\") pod \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\" (UID: \"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed\") "
Jul 09 10:05:45.964272 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.964252    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" (UID: "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:05:45.964347 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.964332    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:45.965930 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.965908    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" (UID: "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:05:45.966020 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:45.965984    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl" (OuterVolumeSpecName: "kube-api-access-jddbl") pod "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" (UID: "15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed"). InnerVolumeSpecName "kube-api-access-jddbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:05:46.065523 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.065452    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddbl\" (UniqueName: \"kubernetes.io/projected/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-kube-api-access-jddbl\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:46.065523 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.065477    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:46.203029 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.202993    2576 generic.go:356] "Generic (PLEG): container finished" podID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerID="b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3" exitCode=0
Jul 09 10:05:46.203492 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.203055    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"
Jul 09 10:05:46.203492 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.203087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerDied","Data":"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"}
Jul 09 10:05:46.203492 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.203164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq" event={"ID":"15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed","Type":"ContainerDied","Data":"595ce4d61c5558341de1be1f6b0a7e6f404ca488b59f0dd59eaf97a99ad3d9ca"}
Jul 09 10:05:46.203492 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.203217    2576 scope.go:117] "RemoveContainer" containerID="f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"
Jul 09 10:05:46.203729 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.203704    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:05:46.212350 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.212334    2576 scope.go:117] "RemoveContainer" containerID="b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"
Jul 09 10:05:46.219364 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.219347    2576 scope.go:117] "RemoveContainer" containerID="f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"
Jul 09 10:05:46.219583 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:46.219563    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939\": container with ID starting with f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939 not found: ID does not exist" containerID="f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"
Jul 09 10:05:46.219623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.219592    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939"} err="failed to get container status \"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939\": rpc error: code = NotFound desc = could not find container \"f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939\": container with ID starting with f04c356f81319ae7449d62e495297c261d4d8bf1c599810d29dbf359f34f2939 not found: ID does not exist"
Jul 09 10:05:46.219623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.219608    2576 scope.go:117] "RemoveContainer" containerID="b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"
Jul 09 10:05:46.219860 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:46.219844    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3\": container with ID starting with b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3 not found: ID does not exist" containerID="b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"
Jul 09 10:05:46.219894 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.219865    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3"} err="failed to get container status \"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3\": rpc error: code = NotFound desc = could not find container \"b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3\": container with ID starting with b963bbc56530e91666e47f2778cdcc2226dd6127c4adb5754062c5827c0e77a3 not found: ID does not exist"
Jul 09 10:05:46.224878 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.224854    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:05:46.228642 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:46.228621    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d678-predictor-6b97f5f755-bbpfq"]
Jul 09 10:05:47.138609 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:47.138572    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:05:47.542206 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:47.542158    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" path="/var/lib/kubelet/pods/15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed/volumes"
Jul 09 10:05:49.149981 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:49.149941    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:50.895572 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:50.895527    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:51.212395 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:51.212364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:05:51.212872 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:51.212845    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:05:54.150048 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:54.150007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:54.150539 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:54.150118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:05:55.895096 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:55.895056    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:57.139394 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:57.139360    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:05:58.523805 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.523784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 10:05:58.570491 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.570468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls\") pod \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") "
Jul 09 10:05:58.570608 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.570503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle\") pod \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\" (UID: \"abc57c84-5d6a-4fdc-afe6-c359baf509ec\") "
Jul 09 10:05:58.570963 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.570936    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "abc57c84-5d6a-4fdc-afe6-c359baf509ec" (UID: "abc57c84-5d6a-4fdc-afe6-c359baf509ec"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:05:58.572413 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.572394    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abc57c84-5d6a-4fdc-afe6-c359baf509ec" (UID: "abc57c84-5d6a-4fdc-afe6-c359baf509ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:05:58.671788 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.671725    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc57c84-5d6a-4fdc-afe6-c359baf509ec-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:58.671788 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:58.671748    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc57c84-5d6a-4fdc-afe6-c359baf509ec-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:05:59.149825 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.149790    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:05:59.264235 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.264205    2576 generic.go:356] "Generic (PLEG): container finished" podID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerID="0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859" exitCode=0
Jul 09 10:05:59.264385 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.264243    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"
Jul 09 10:05:59.264385 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.264243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" event={"ID":"abc57c84-5d6a-4fdc-afe6-c359baf509ec","Type":"ContainerDied","Data":"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"}
Jul 09 10:05:59.264385 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.264362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4" event={"ID":"abc57c84-5d6a-4fdc-afe6-c359baf509ec","Type":"ContainerDied","Data":"cf8ba804ca9060c68c4bd34b6c660d3a8e7221099e833d541f8b7e02fbc0b7d4"}
Jul 09 10:05:59.264483 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.264388    2576 scope.go:117] "RemoveContainer" containerID="0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"
Jul 09 10:05:59.272761 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.272745    2576 scope.go:117] "RemoveContainer" containerID="0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"
Jul 09 10:05:59.273015 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:05:59.272995    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859\": container with ID starting with 0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859 not found: ID does not exist" containerID="0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"
Jul 09 10:05:59.273064 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.273025    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859"} err="failed to get container status \"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859\": rpc error: code = NotFound desc = could not find container \"0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859\": container with ID starting with 0651b67e4a37373aaddd80d2957586592fa74a15acd7ddbd8d92de87b1c63859 not found: ID does not exist"
Jul 09 10:05:59.285513 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.285494    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 10:05:59.290415 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.290396    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2cd6b-748d5dcdf4-8jbw4"]
Jul 09 10:05:59.542633 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:05:59.542601    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" path="/var/lib/kubelet/pods/abc57c84-5d6a-4fdc-afe6-c359baf509ec/volumes"
Jul 09 10:06:01.213017 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:01.212973    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:06:04.149510 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:04.149473    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:06:07.138938 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:07.138902    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 10:06:09.149768 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:09.149728    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:06:11.213540 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:11.213505    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:06:12.859591 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:12.859567    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:06:12.988012 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:12.987933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle\") pod \"5185756b-f66b-43b8-abec-4422ec5c999e\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") "
Jul 09 10:06:12.988158 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:12.988019    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls\") pod \"5185756b-f66b-43b8-abec-4422ec5c999e\" (UID: \"5185756b-f66b-43b8-abec-4422ec5c999e\") "
Jul 09 10:06:12.988272 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:12.988251    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5185756b-f66b-43b8-abec-4422ec5c999e" (UID: "5185756b-f66b-43b8-abec-4422ec5c999e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:06:12.989913 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:12.989888    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5185756b-f66b-43b8-abec-4422ec5c999e" (UID: "5185756b-f66b-43b8-abec-4422ec5c999e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:06:13.088665 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.088629    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5185756b-f66b-43b8-abec-4422ec5c999e-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:06:13.088665 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.088664    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5185756b-f66b-43b8-abec-4422ec5c999e-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:06:13.325176 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.325085    2576 generic.go:356] "Generic (PLEG): container finished" podID="5185756b-f66b-43b8-abec-4422ec5c999e" containerID="b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e" exitCode=137
Jul 09 10:06:13.325176 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.325136    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"
Jul 09 10:06:13.325176 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.325151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" event={"ID":"5185756b-f66b-43b8-abec-4422ec5c999e","Type":"ContainerDied","Data":"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"}
Jul 09 10:06:13.325462 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.325207    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49" event={"ID":"5185756b-f66b-43b8-abec-4422ec5c999e","Type":"ContainerDied","Data":"dee8dc9f10d3c0e2aa99fb7e461e74f0451ef4638236bd55aaa5c29cb65201fb"}
Jul 09 10:06:13.325462 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.325230    2576 scope.go:117] "RemoveContainer" containerID="b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"
Jul 09 10:06:13.333975 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.333954    2576 scope.go:117] "RemoveContainer" containerID="b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"
Jul 09 10:06:13.334238 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:06:13.334218    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e\": container with ID starting with b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e not found: ID does not exist" containerID="b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"
Jul 09 10:06:13.334296 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.334249    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e"} err="failed to get container status \"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e\": rpc error: code = NotFound desc = could not find container \"b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e\": container with ID starting with b962d72c05d50938771ed8547e102191cd10e1f00b3b9daa220bb4a298e7254e not found: ID does not exist"
Jul 09 10:06:13.346915 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.346892    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:06:13.352489 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.352468    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d678-56c66cc487-c4m49"]
Jul 09 10:06:13.541074 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:13.541041    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" path="/var/lib/kubelet/pods/5185756b-f66b-43b8-abec-4422ec5c999e/volumes"
Jul 09 10:06:17.139385 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:17.139351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:06:21.213750 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:21.213703    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 10:06:28.633976 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.633903    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634281    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634293    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634302    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634307    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634320    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634326    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634340    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kube-rbac-proxy"
Jul 09 10:06:28.634354 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634345    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kube-rbac-proxy"
Jul 09 10:06:28.634600 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634399    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5185756b-f66b-43b8-abec-4422ec5c999e" containerName="splitter-graph-9d678"
Jul 09 10:06:28.634600 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634407    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc57c84-5d6a-4fdc-afe6-c359baf509ec" containerName="sequence-graph-2cd6b"
Jul 09 10:06:28.634600 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634414    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kube-rbac-proxy"
Jul 09 10:06:28.634600 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.634421    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e4fa1a-11e6-4f3a-8571-12fe9ea6f9ed" containerName="kserve-container"
Jul 09 10:06:28.638784 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.638766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.641402 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.641382    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-52f9b-kube-rbac-proxy-sar-config\""
Jul 09 10:06:28.641523 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.641384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-52f9b-serving-cert\""
Jul 09 10:06:28.649899 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.649875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:06:28.650002 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.649924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:06:28.813313 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.813269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.813313 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.813316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.888449 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.888355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:06:28.914073 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.914041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.914259 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.914080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.914843 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.914819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.916560 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.916539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls\") pod \"switch-graph-52f9b-557dd99dc5-s2n4c\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") " pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:28.948595 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:28.948568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:29.077281 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:29.077249    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:06:29.080454 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:06:29.080429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce8c242_90c3_4b50_8ebe_e51dbdf7a478.slice/crio-64a92740725a287c1502947379159dee26420c1f6e492a52f49ef12b91758eb6 WatchSource:0}: Error finding container 64a92740725a287c1502947379159dee26420c1f6e492a52f49ef12b91758eb6: Status 404 returned error can't find the container with id 64a92740725a287c1502947379159dee26420c1f6e492a52f49ef12b91758eb6
Jul 09 10:06:29.398537 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:29.398504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" event={"ID":"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478","Type":"ContainerStarted","Data":"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"}
Jul 09 10:06:29.398698 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:29.398555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" event={"ID":"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478","Type":"ContainerStarted","Data":"64a92740725a287c1502947379159dee26420c1f6e492a52f49ef12b91758eb6"}
Jul 09 10:06:29.398698 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:29.398685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:29.416291 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:29.416246    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podStartSLOduration=1.416232715 podStartE2EDuration="1.416232715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:06:29.415506824 +0000 UTC m=+1974.451316530" watchObservedRunningTime="2026-07-09 10:06:29.416232715 +0000 UTC m=+1974.452042420"
Jul 09 10:06:31.214236 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:31.214209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:06:35.408576 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:35.408546    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:06:42.951852 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.951820    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:06:42.955923 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.955902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:42.957056 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.957029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:06:42.958770 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.958750    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a7fc2-serving-cert\""
Jul 09 10:06:42.959065 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.959050    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a7fc2-kube-rbac-proxy-sar-config\""
Jul 09 10:06:42.967043 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:42.967026    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:06:43.027790 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.027765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.027882 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.027805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.128762 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.128725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.128893 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.128825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.128936 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:06:43.128893    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-a7fc2-serving-cert: secret "splitter-graph-a7fc2-serving-cert" not found
Jul 09 10:06:43.128974 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:06:43.128957    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls podName:9df3cb7b-0fe3-4690-8692-c19f344ef5a6 nodeName:}" failed. No retries permitted until 2026-07-09 10:06:43.628936727 +0000 UTC m=+1988.664746412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls") pod "splitter-graph-a7fc2-5d85f6c84f-hxbwt" (UID: "9df3cb7b-0fe3-4690-8692-c19f344ef5a6") : secret "splitter-graph-a7fc2-serving-cert" not found
Jul 09 10:06:43.129495 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.129477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.242251 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.242170    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:06:43.632062 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.631975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.634512 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.634488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") pod \"splitter-graph-a7fc2-5d85f6c84f-hxbwt\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") " pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.867798 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.867746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:43.999296 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:43.999267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:06:44.002055 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:06:44.002029    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df3cb7b_0fe3_4690_8692_c19f344ef5a6.slice/crio-86240517523ace6019f2d81c8e6406662592f133f5e3ca70a42ad63673060005 WatchSource:0}: Error finding container 86240517523ace6019f2d81c8e6406662592f133f5e3ca70a42ad63673060005: Status 404 returned error can't find the container with id 86240517523ace6019f2d81c8e6406662592f133f5e3ca70a42ad63673060005
Jul 09 10:06:44.469477 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:44.469441    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" event={"ID":"9df3cb7b-0fe3-4690-8692-c19f344ef5a6","Type":"ContainerStarted","Data":"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"}
Jul 09 10:06:44.469638 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:44.469494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" event={"ID":"9df3cb7b-0fe3-4690-8692-c19f344ef5a6","Type":"ContainerStarted","Data":"86240517523ace6019f2d81c8e6406662592f133f5e3ca70a42ad63673060005"}
Jul 09 10:06:44.469638 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:44.469526    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:06:44.488039 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:44.487976    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podStartSLOduration=2.487958185 podStartE2EDuration="2.487958185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:06:44.487437739 +0000 UTC m=+1989.523247446" watchObservedRunningTime="2026-07-09 10:06:44.487958185 +0000 UTC m=+1989.523767890"
Jul 09 10:06:50.479850 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:06:50.479817    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:11:56.683709 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:11:56.683677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:12:35.401567 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:12:35.401482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:12:44.581803 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:12:44.581763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:12:44.639046 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:12:44.639007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:14:57.596254 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.596222    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:14:57.598569 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.596410    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" containerID="cri-o://239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb" gracePeriod=30
Jul 09 10:14:57.666644 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.666615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:14:57.914197 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.914098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:14:57.970155 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.970125    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:14:57.970359 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.970338    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container" containerID="cri-o://812adecf8bb786e80f154b60c44216b1ca9eb8ef31536b2ac0ced2391fdf1745" gracePeriod=30
Jul 09 10:14:57.970461 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:57.970413    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kube-rbac-proxy" containerID="cri-o://3d373704f9b72a41337a7626140ec3576f588da8ee850f818e2460008a0d09e6" gracePeriod=30
Jul 09 10:14:58.005569 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:58.005549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:14:58.655124 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:58.655088    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerID="3d373704f9b72a41337a7626140ec3576f588da8ee850f818e2460008a0d09e6" exitCode=2
Jul 09 10:14:58.655530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:14:58.655166    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerDied","Data":"3d373704f9b72a41337a7626140ec3576f588da8ee850f818e2460008a0d09e6"}
Jul 09 10:15:00.478399 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.478363    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:00.666414 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.666384    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerID="812adecf8bb786e80f154b60c44216b1ca9eb8ef31536b2ac0ced2391fdf1745" exitCode=0
Jul 09 10:15:00.666550 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.666456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerDied","Data":"812adecf8bb786e80f154b60c44216b1ca9eb8ef31536b2ac0ced2391fdf1745"}
Jul 09 10:15:00.705966 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.705947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:15:00.844327 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.844267    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config\") pod \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") "
Jul 09 10:15:00.844327 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.844302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbvq4\" (UniqueName: \"kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4\") pod \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") "
Jul 09 10:15:00.844477 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.844339    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") pod \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\" (UID: \"3a6481a6-4e72-4e32-adb4-d4a9932d530c\") "
Jul 09 10:15:00.844684 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.844661    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a6481a6-4e72-4e32-adb4-d4a9932d530c" (UID: "3a6481a6-4e72-4e32-adb4-d4a9932d530c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:15:00.846345 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.846315    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4" (OuterVolumeSpecName: "kube-api-access-jbvq4") pod "3a6481a6-4e72-4e32-adb4-d4a9932d530c" (UID: "3a6481a6-4e72-4e32-adb4-d4a9932d530c"). InnerVolumeSpecName "kube-api-access-jbvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:15:00.846345 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.846318    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a6481a6-4e72-4e32-adb4-d4a9932d530c" (UID: "3a6481a6-4e72-4e32-adb4-d4a9932d530c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:15:00.945039 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.945015    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:15:00.945039 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.945039    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvq4\" (UniqueName: \"kubernetes.io/projected/3a6481a6-4e72-4e32-adb4-d4a9932d530c-kube-api-access-jbvq4\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:15:00.945227 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:00.945049    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6481a6-4e72-4e32-adb4-d4a9932d530c-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:15:01.675939 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.675905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs" event={"ID":"3a6481a6-4e72-4e32-adb4-d4a9932d530c","Type":"ContainerDied","Data":"cec37fda1f3898b138e6414f38626d8f19f34355c995e79df02aa811d4347cf2"}
Jul 09 10:15:01.676397 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.675950    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"
Jul 09 10:15:01.676397 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.675978    2576 scope.go:117] "RemoveContainer" containerID="3d373704f9b72a41337a7626140ec3576f588da8ee850f818e2460008a0d09e6"
Jul 09 10:15:01.684333 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.684311    2576 scope.go:117] "RemoveContainer" containerID="812adecf8bb786e80f154b60c44216b1ca9eb8ef31536b2ac0ced2391fdf1745"
Jul 09 10:15:01.694296 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.694273    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:15:01.698596 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:01.698574    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a7fc2-predictor-78f845d769-bmbbs"]
Jul 09 10:15:03.541338 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:03.541298    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" path="/var/lib/kubelet/pods/3a6481a6-4e72-4e32-adb4-d4a9932d530c/volumes"
Jul 09 10:15:05.478348 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:05.478303    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:10.477548 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:10.477504    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:10.478039 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:10.477613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:15:15.478311 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:15.478277    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:20.478351 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:20.478303    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:25.477784 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:25.477733    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:15:27.746484 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.746462    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:15:27.790549 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.790524    2576 generic.go:356] "Generic (PLEG): container finished" podID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerID="239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb" exitCode=0
Jul 09 10:15:27.790680 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.790562    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"
Jul 09 10:15:27.790680 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.790581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" event={"ID":"9df3cb7b-0fe3-4690-8692-c19f344ef5a6","Type":"ContainerDied","Data":"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"}
Jul 09 10:15:27.790680 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.790621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt" event={"ID":"9df3cb7b-0fe3-4690-8692-c19f344ef5a6","Type":"ContainerDied","Data":"86240517523ace6019f2d81c8e6406662592f133f5e3ca70a42ad63673060005"}
Jul 09 10:15:27.790680 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.790642    2576 scope.go:117] "RemoveContainer" containerID="239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"
Jul 09 10:15:27.798628 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.798613    2576 scope.go:117] "RemoveContainer" containerID="239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"
Jul 09 10:15:27.798882 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:15:27.798862    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb\": container with ID starting with 239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb not found: ID does not exist" containerID="239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"
Jul 09 10:15:27.798924 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.798890    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb"} err="failed to get container status \"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb\": rpc error: code = NotFound desc = could not find container \"239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb\": container with ID starting with 239a386f0c8337961db228cafda41f66a4fa24d2e123b432c5c4798d3b6b78fb not found: ID does not exist"
Jul 09 10:15:27.850295 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.850248    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") pod \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") "
Jul 09 10:15:27.850373 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.850308    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle\") pod \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\" (UID: \"9df3cb7b-0fe3-4690-8692-c19f344ef5a6\") "
Jul 09 10:15:27.850645 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.850621    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9df3cb7b-0fe3-4690-8692-c19f344ef5a6" (UID: "9df3cb7b-0fe3-4690-8692-c19f344ef5a6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:15:27.852096 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.852078    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9df3cb7b-0fe3-4690-8692-c19f344ef5a6" (UID: "9df3cb7b-0fe3-4690-8692-c19f344ef5a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:15:27.951516 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.951494    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:15:27.951516 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:27.951514    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df3cb7b-0fe3-4690-8692-c19f344ef5a6-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:15:28.112447 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:28.112380    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:15:28.117924 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:28.117901    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a7fc2-5d85f6c84f-hxbwt"]
Jul 09 10:15:29.541289 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:15:29.541259    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" path="/var/lib/kubelet/pods/9df3cb7b-0fe3-4690-8692-c19f344ef5a6/volumes"
Jul 09 10:18:37.782956 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:18:37.782879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:20:44.133589 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:20:44.133556    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:47.913293 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:47.912752    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:22:47.913293 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:47.913079    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" containerID="cri-o://f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1" gracePeriod=30
Jul 09 10:22:47.974925 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:47.974887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:48.101032 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.100999    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:48.133494 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.133458    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:22:48.133766 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.133736    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container" containerID="cri-o://c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4" gracePeriod=30
Jul 09 10:22:48.133861 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.133743    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kube-rbac-proxy" containerID="cri-o://ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52" gracePeriod=30
Jul 09 10:22:48.157877 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.157835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:48.668206 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.668090    2576 generic.go:356] "Generic (PLEG): container finished" podID="87a70fe9-fdf0-43ef-a105-083311069702" containerID="ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52" exitCode=2
Jul 09 10:22:48.668206 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.668152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerDied","Data":"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"}
Jul 09 10:22:48.890710 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.890678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:48.989301 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:48.989269    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:22:50.407469 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:50.407432    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:22:51.079700 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.079680    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:22:51.134405 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.134380    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config\") pod \"87a70fe9-fdf0-43ef-a105-083311069702\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") "
Jul 09 10:22:51.134533 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.134426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp5d7\" (UniqueName: \"kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7\") pod \"87a70fe9-fdf0-43ef-a105-083311069702\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") "
Jul 09 10:22:51.134533 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.134458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls\") pod \"87a70fe9-fdf0-43ef-a105-083311069702\" (UID: \"87a70fe9-fdf0-43ef-a105-083311069702\") "
Jul 09 10:22:51.134770 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.134747    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87a70fe9-fdf0-43ef-a105-083311069702" (UID: "87a70fe9-fdf0-43ef-a105-083311069702"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:22:51.136430 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.136408    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87a70fe9-fdf0-43ef-a105-083311069702" (UID: "87a70fe9-fdf0-43ef-a105-083311069702"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:22:51.136548 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.136530    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7" (OuterVolumeSpecName: "kube-api-access-bp5d7") pod "87a70fe9-fdf0-43ef-a105-083311069702" (UID: "87a70fe9-fdf0-43ef-a105-083311069702"). InnerVolumeSpecName "kube-api-access-bp5d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:22:51.236066 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.236006    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a70fe9-fdf0-43ef-a105-083311069702-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:22:51.236066 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.236029    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5d7\" (UniqueName: \"kubernetes.io/projected/87a70fe9-fdf0-43ef-a105-083311069702-kube-api-access-bp5d7\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:22:51.236066 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.236039    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a70fe9-fdf0-43ef-a105-083311069702-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:22:51.682167 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.682131    2576 generic.go:356] "Generic (PLEG): container finished" podID="87a70fe9-fdf0-43ef-a105-083311069702" containerID="c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4" exitCode=0
Jul 09 10:22:51.682589 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.682196    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"
Jul 09 10:22:51.682589 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.682206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerDied","Data":"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"}
Jul 09 10:22:51.682589 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.682264    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg" event={"ID":"87a70fe9-fdf0-43ef-a105-083311069702","Type":"ContainerDied","Data":"5424fedcc9c49950826a84dfa5d373b81810a8982f93840c386f42da18d098b3"}
Jul 09 10:22:51.682589 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.682293    2576 scope.go:117] "RemoveContainer" containerID="ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"
Jul 09 10:22:51.690530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.690508    2576 scope.go:117] "RemoveContainer" containerID="c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"
Jul 09 10:22:51.697864 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.697810    2576 scope.go:117] "RemoveContainer" containerID="ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"
Jul 09 10:22:51.698147 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:22:51.698118    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52\": container with ID starting with ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52 not found: ID does not exist" containerID="ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"
Jul 09 10:22:51.698257 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.698158    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52"} err="failed to get container status \"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52\": rpc error: code = NotFound desc = could not find container \"ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52\": container with ID starting with ad712fa42377fb509987ac8d09bca731797db9afbadaa60242604df00d7d5c52 not found: ID does not exist"
Jul 09 10:22:51.698257 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.698198    2576 scope.go:117] "RemoveContainer" containerID="c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"
Jul 09 10:22:51.698487 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:22:51.698463    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4\": container with ID starting with c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4 not found: ID does not exist" containerID="c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"
Jul 09 10:22:51.698530 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.698496    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4"} err="failed to get container status \"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4\": rpc error: code = NotFound desc = could not find container \"c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4\": container with ID starting with c298ec650f6196be3463e2fa96eb4b1e3c9c2d8c6512301794ea2d5733d56cf4 not found: ID does not exist"
Jul 09 10:22:51.699788 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.699770    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:22:51.703137 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:51.703118    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52f9b-predictor-78498cf5bc-mcbjg"]
Jul 09 10:22:53.541085 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:53.541052    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a70fe9-fdf0-43ef-a105-083311069702" path="/var/lib/kubelet/pods/87a70fe9-fdf0-43ef-a105-083311069702/volumes"
Jul 09 10:22:55.407034 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:22:55.406988    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:23:00.407953 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:00.407915    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:23:00.408421 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:00.408041    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:23:04.210932 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:04.210897    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:05.031457 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:05.031407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:05.407265 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:05.407136    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:23:05.920909 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:05.920879    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:06.826973 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:06.826938    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:07.611258 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:07.611226    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:08.418544 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:08.418515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:09.235309 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:09.235267    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:10.031496 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:10.031463    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:10.407820 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:10.407729    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:23:10.846914 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:10.846877    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:11.622210 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:11.622169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:12.395063 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:12.395034    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:13.171474 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:13.171428    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:13.939595 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:13.939560    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:14.743005 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:14.742962    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-52f9b-557dd99dc5-s2n4c_2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/switch-graph-52f9b/0.log"
Jul 09 10:23:15.407362 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:15.407322    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 10:23:18.070326 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.070298    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:23:18.251892 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.251868    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle\") pod \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") "
Jul 09 10:23:18.252045 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.251920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls\") pod \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\" (UID: \"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478\") "
Jul 09 10:23:18.252174 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.252151    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" (UID: "2ce8c242-90c3-4b50-8ebe-e51dbdf7a478"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:23:18.253884 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.253864    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" (UID: "2ce8c242-90c3-4b50-8ebe-e51dbdf7a478"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:23:18.353131 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.353102    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-proxy-tls\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:23:18.353131 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.353128    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478-openshift-service-ca-bundle\") on node \"ip-10-0-141-211.ec2.internal\" DevicePath \"\""
Jul 09 10:23:18.796766 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.796736    2576 generic.go:356] "Generic (PLEG): container finished" podID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerID="f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1" exitCode=0
Jul 09 10:23:18.797050 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.796796    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"
Jul 09 10:23:18.797050 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.796817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" event={"ID":"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478","Type":"ContainerDied","Data":"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"}
Jul 09 10:23:18.797050 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.796878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c" event={"ID":"2ce8c242-90c3-4b50-8ebe-e51dbdf7a478","Type":"ContainerDied","Data":"64a92740725a287c1502947379159dee26420c1f6e492a52f49ef12b91758eb6"}
Jul 09 10:23:18.797050 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.796910    2576 scope.go:117] "RemoveContainer" containerID="f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"
Jul 09 10:23:18.805262 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.805246    2576 scope.go:117] "RemoveContainer" containerID="f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"
Jul 09 10:23:18.805538 ip-10-0-141-211 kubenswrapper[2576]: E0709 10:23:18.805520    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1\": container with ID starting with f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1 not found: ID does not exist" containerID="f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"
Jul 09 10:23:18.805579 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.805547    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1"} err="failed to get container status \"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1\": rpc error: code = NotFound desc = could not find container \"f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1\": container with ID starting with f4f7a2c85ec21e9cae6244c51ccc794e53cb9b3454047640f0365a336d4248c1 not found: ID does not exist"
Jul 09 10:23:18.817272 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.817252    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:23:18.821996 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:18.821972    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-52f9b-557dd99dc5-s2n4c"]
Jul 09 10:23:19.541816 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:19.541783    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" path="/var/lib/kubelet/pods/2ce8c242-90c3-4b50-8ebe-e51dbdf7a478/volumes"
Jul 09 10:23:20.381307 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:20.381273    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:23:21.336012 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:21.335975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:23:21.409370 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:21.409338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:23:24.320697 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:24.320665    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-29jzx_021994c4-fc8b-4b9c-b154-569016c95dfe/global-pull-secret-syncer/0.log"
Jul 09 10:23:24.527581 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:24.527554    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log"
Jul 09 10:23:24.646408 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:24.646324    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-211.ec2.internal_747907e14aa1fadee2058ca31d2de152/haproxy/0.log"
Jul 09 10:23:26.044080 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:26.044034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:23:26.107508 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:26.107473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fsw2t_0df0c049-1ee6-423f-9523-2b81b60e9b71/konnectivity-agent/0.log" limit=1024
Jul 09 10:23:27.800553 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.800520    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/alertmanager/0.log"
Jul 09 10:23:27.826231 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.826169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/config-reloader/0.log"
Jul 09 10:23:27.847854 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.847827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/kube-rbac-proxy-web/0.log"
Jul 09 10:23:27.883226 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.883196    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/kube-rbac-proxy/0.log"
Jul 09 10:23:27.910607 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.910580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/kube-rbac-proxy-metric/0.log"
Jul 09 10:23:27.939601 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.939580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/prom-label-proxy/0.log"
Jul 09 10:23:27.971004 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:27.970980    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ea86ffd-adcb-4790-9571-7354d5e752a3/init-config-reloader/0.log"
Jul 09 10:23:28.164475 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.164414    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-758f5fb9d-bz8j2_c3b2dc96-3727-45dd-aeec-795763c790b9/metrics-server/0.log"
Jul 09 10:23:28.245003 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.244979    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7664_3b227347-b1fd-456a-aa5f-5ad529d0cb35/node-exporter/0.log"
Jul 09 10:23:28.274200 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.274158    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7664_3b227347-b1fd-456a-aa5f-5ad529d0cb35/kube-rbac-proxy/0.log"
Jul 09 10:23:28.302603 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.302584    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7664_3b227347-b1fd-456a-aa5f-5ad529d0cb35/init-textfile/0.log"
Jul 09 10:23:28.552393 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.552368    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhwz8_c2422981-b620-4ecc-b9ae-9f4fa0ef5f97/kube-rbac-proxy-main/0.log"
Jul 09 10:23:28.587578 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.587548    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhwz8_c2422981-b620-4ecc-b9ae-9f4fa0ef5f97/kube-rbac-proxy-self/0.log"
Jul 09 10:23:28.623034 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:28.622985    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhwz8_c2422981-b620-4ecc-b9ae-9f4fa0ef5f97/openshift-state-metrics/0.log"
Jul 09 10:23:29.108221 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.108154    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7c4c975c-6rvkn_91dea31f-f28c-41f3-9807-a5c0e47a8d49/telemeter-client/0.log"
Jul 09 10:23:29.142977 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.142952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7c4c975c-6rvkn_91dea31f-f28c-41f3-9807-a5c0e47a8d49/reload/0.log"
Jul 09 10:23:29.168519 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.168483    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7c4c975c-6rvkn_91dea31f-f28c-41f3-9807-a5c0e47a8d49/kube-rbac-proxy/0.log"
Jul 09 10:23:29.199561 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.199530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/thanos-query/0.log"
Jul 09 10:23:29.222524 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.222491    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/kube-rbac-proxy-web/0.log"
Jul 09 10:23:29.247192 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.247163    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/kube-rbac-proxy/0.log"
Jul 09 10:23:29.269150 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.269129    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/prom-label-proxy/0.log"
Jul 09 10:23:29.293075 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.293053    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/kube-rbac-proxy-rules/0.log"
Jul 09 10:23:29.314742 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:29.314721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94785f998-rpp9k_667749e1-7115-4226-b246-1af1a3fde4e5/kube-rbac-proxy-metrics/0.log"
Jul 09 10:23:32.848137 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848101    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"]
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848477    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848493    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848501    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848507    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848514    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b"
Jul 09 10:23:32.848529 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848521    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848540    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848547    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848555    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848560    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848571    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848576    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848629    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df3cb7b-0fe3-4690-8692-c19f344ef5a6" containerName="splitter-graph-a7fc2"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848638    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce8c242-90c3-4b50-8ebe-e51dbdf7a478" containerName="switch-graph-52f9b"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848651    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6481a6-4e72-4e32-adb4-d4a9932d530c" containerName="kserve-container"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848656    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kube-rbac-proxy"
Jul 09 10:23:32.848720 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.848662    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a70fe9-fdf0-43ef-a105-083311069702" containerName="kserve-container"
Jul 09 10:23:32.853011 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.852986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.854670 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.854643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-lib-modules\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.854779 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.854685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptqp\" (UniqueName: \"kubernetes.io/projected/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-kube-api-access-6ptqp\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.854779 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.854735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-sys\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.854779 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.854772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-podres\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.854934 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.854793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-proc\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.855668 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.855643    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8qv5\"/\"kube-root-ca.crt\""
Jul 09 10:23:32.855668 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.855664    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q8qv5\"/\"default-dockercfg-4jsqr\""
Jul 09 10:23:32.855840 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.855664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8qv5\"/\"openshift-service-ca.crt\""
Jul 09 10:23:32.862215 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.862173    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"]
Jul 09 10:23:32.955414 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-lib-modules\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955414 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptqp\" (UniqueName: \"kubernetes.io/projected/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-kube-api-access-6ptqp\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-sys\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-podres\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-proc\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955509    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-lib-modules\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955545    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-sys\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955559    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-proc\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.955623 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.955584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-podres\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:32.965400 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:32.965373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptqp\" (UniqueName: \"kubernetes.io/projected/ce66ff6a-1e1a-4d0f-b3a7-a709889716ee-kube-api-access-6ptqp\") pod \"perf-node-gather-daemonset-xc6z2\" (UID: \"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:33.114811 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.114727    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbr8d_c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329/dns/0.log"
Jul 09 10:23:33.134691 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.134669    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbr8d_c84ae4b3-3ecf-4ab6-ac3c-348fb7b12329/kube-rbac-proxy/0.log"
Jul 09 10:23:33.163920 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.163895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:33.314923 ip-10-0-141-211 kubenswrapper[2576]: W0709 10:23:33.314892    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce66ff6a_1e1a_4d0f_b3a7_a709889716ee.slice/crio-a526ce8ab66aa36261d873bf1e6568f1ec2849ae3c7c665c5c7a7617a0ca166c WatchSource:0}: Error finding container a526ce8ab66aa36261d873bf1e6568f1ec2849ae3c7c665c5c7a7617a0ca166c: Status 404 returned error can't find the container with id a526ce8ab66aa36261d873bf1e6568f1ec2849ae3c7c665c5c7a7617a0ca166c
Jul 09 10:23:33.315223 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.315199    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"]
Jul 09 10:23:33.346877 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.346851    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpdj2_fd672333-421c-432f-8bc5-972576520eca/dns-node-resolver/0.log"
Jul 09 10:23:33.840151 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.840120    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9547fb7d9-5vjpd_10f18fc5-6c98-407f-a197-46ad302411d2/registry/0.log"
Jul 09 10:23:33.862269 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.862239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2" event={"ID":"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee","Type":"ContainerStarted","Data":"9976b824a8e330bc4ce05166600f8da180f5ab5cccd0b0abb45e51afee4edafa"}
Jul 09 10:23:33.862619 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.862284    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2" event={"ID":"ce66ff6a-1e1a-4d0f-b3a7-a709889716ee","Type":"ContainerStarted","Data":"a526ce8ab66aa36261d873bf1e6568f1ec2849ae3c7c665c5c7a7617a0ca166c"}
Jul 09 10:23:33.862619 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.862364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:33.886644 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.886602    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2" podStartSLOduration=1.8865893379999998 podStartE2EDuration="1.886589338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:23:33.885230759 +0000 UTC m=+2998.921040464" watchObservedRunningTime="2026-07-09 10:23:33.886589338 +0000 UTC m=+2998.922399079"
Jul 09 10:23:33.917009 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:33.916986    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jcl8t_719992de-6f8d-4661-bd86-c31a9f12e6b5/node-ca/0.log"
Jul 09 10:23:35.272527 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:35.272501    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8wsv_66d45501-b2b0-4850-acbe-ba3f3bf2a835/serve-healthcheck-canary/0.log"
Jul 09 10:23:35.774062 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:35.774036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glzrj_66291734-ad8e-470c-9aa9-1069950ad995/kube-rbac-proxy/0.log"
Jul 09 10:23:35.797743 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:35.797717    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glzrj_66291734-ad8e-470c-9aa9-1069950ad995/exporter/0.log"
Jul 09 10:23:35.828243 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:35.828217    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glzrj_66291734-ad8e-470c-9aa9-1069950ad995/extractor/0.log"
Jul 09 10:23:38.056851 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:38.056820    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5876b4f96d-htq6x_55cb3d18-767c-44a4-b768-11c9acaa6545/manager/0.log"
Jul 09 10:23:38.090922 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:38.090895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-rqnv4_c9d2a769-a344-4c12-bd9b-0259fa8d0748/manager/0.log"
Jul 09 10:23:38.265191 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:38.265150    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-6g2nc_34aa4be3-2a02-441a-90ea-a64af040e0de/manager/0.log"
Jul 09 10:23:39.875883 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:39.875849    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-xc6z2"
Jul 09 10:23:43.809367 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.809340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/kube-multus-additional-cni-plugins/0.log"
Jul 09 10:23:43.833390 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.833366    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/egress-router-binary-copy/0.log"
Jul 09 10:23:43.857284 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.857262    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/cni-plugins/0.log"
Jul 09 10:23:43.877996 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.877972    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/bond-cni-plugin/0.log"
Jul 09 10:23:43.898248 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.898230    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/routeoverride-cni/0.log"
Jul 09 10:23:43.919133 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.919118    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/whereabouts-cni-bincopy/0.log"
Jul 09 10:23:43.940768 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:43.940748    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b58d9_471543cd-d0be-4b9a-8c44-b9d21c91a3a5/whereabouts-cni/0.log"
Jul 09 10:23:44.422386 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:44.422293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wghbx_8b33451b-4bff-453a-b2c4-0409582be7c7/kube-multus/0.log"
Jul 09 10:23:44.487065 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:44.487036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ws58m_53ce9f2d-7e9f-4933-aa70-236be2d8b270/network-metrics-daemon/0.log"
Jul 09 10:23:44.503339 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:44.503314    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ws58m_53ce9f2d-7e9f-4933-aa70-236be2d8b270/kube-rbac-proxy/0.log"
Jul 09 10:23:45.507443 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.507412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/ovn-controller/0.log"
Jul 09 10:23:45.530942 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.530881    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/ovn-acl-logging/0.log"
Jul 09 10:23:45.564999 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.564966    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/kube-rbac-proxy-node/0.log"
Jul 09 10:23:45.586992 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.586967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 10:23:45.607839 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.607815    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/northd/0.log"
Jul 09 10:23:45.627116 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.627095    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/nbdb/0.log"
Jul 09 10:23:45.648659 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.648639    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/sbdb/0.log"
Jul 09 10:23:45.714808 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:45.714783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdmzb_46bd2fa2-68cf-428d-9ef7-7e0ed0d77e5a/ovnkube-controller/0.log"
Jul 09 10:23:47.046710 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:47.046680    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-nlkst_fb2bea31-b084-4006-9ea1-02113a6eeb22/network-check-target-container/0.log"
Jul 09 10:23:48.064167 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:48.064138    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-s7xhc_53793e29-e8b5-4732-acf1-9bc9f65b7ec0/iptables-alerter/0.log"
Jul 09 10:23:48.840732 ip-10-0-141-211 kubenswrapper[2576]: I0709 10:23:48.840702    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-w5ckn_83b54daa-e35f-4061-9647-4abe8e0a9591/tuned/0.log"